Trajectory-Based Viewport Prediction for 360-Degree Videos

ABSTRACT

In implementations of trajectory-based viewport prediction for 360-degree videos, a video system obtains trajectories of angles of users who have previously viewed a 360-degree video. The angles are used to determine viewports of the 360-degree video, and may include trajectories for a yaw angle, a pitch angle, and a roll angle of a user recorded as the user views the 360-degree video. The video system clusters the trajectories of angles into trajectory clusters, and for each trajectory cluster determines a trend trajectory. When a new user views the 360-degree video, the video system compares trajectories of angles of the new user to the trend trajectories, and selects trend trajectories for a yaw angle, a pitch angle, and a roll angle for the user. Using the selected trend trajectories, the video system predicts viewports of the 360-degree video for the user for future times.

RELATED APPLICATIONS

This application claims priority as a continuation of U.S. patentapplication Ser. No. 16/421,276, filed May 23, 2019, and titled“Trajectory-based Viewport Prediction for 360-Degree Videos,” the entiredisclosure of which is hereby incorporated by reference.

BACKGROUND

Videos in which views in multiple directions are simultaneously recorded(e.g., using an omnidirectional camera or multiple cameras) are referredto as 360-degree videos, immersive videos, or spherical videos, and areused in virtual reality, gaming, and playback situations where a viewercan control his or her viewing direction. The part of a 360-degree videobeing viewed by a viewer during playback of the 360-degree video isreferred to as the viewport, and changes as the viewer changes his orher viewing direction. For instance, when playing a video game thatallows a viewer to immerse themselves in the 360-degree video usingvirtual reality, the viewport corresponding to the viewer may change todisplay a different portion of the 360-degree video based on theviewer's movements within the video game.

When delivering a 360-degree video, such as when a server delivers the360-degree video to a client device over a network, the portion of the360-degree video corresponding to a current viewport is often deliveredat a higher quality (e.g., a higher bit-rate of source encoding) thanother portions of the 360-degree video to reduce the bandwidthrequirements needed to deliver the 360-degree video. For instance, the360-degree video can be encoded at different qualities and spatiallydivided into tiles. During the streaming session, the client device canrequest the tiles corresponding to the current viewport at the highestqualities. Consequently, when a user changes the viewport of the360-degree video, such as by moving during playback of the 360-degreevideo, the user may experience a degradation in the video quality at thetransitions of the viewport caused by different encoding qualities ofthe different portions of the 360-degree video. Hence, many videosystems not only request a current viewport for a user at a higherquality, but also predict a future viewport for the user and request thepredicted viewport (e.g., tiles of the 360-degree video corresponding tothe predicted viewport) at a higher quality than other portions of the360-degree video to minimize the transitions in quality experienced bythe user as they change the viewport. This technique is sometimesreferred to as viewport-based adaptive streaming.

Conventional systems that perform viewport-based adaptive streaming arelimited to predicting user viewports for short-term time horizons,typically on the order of milliseconds, and almost always less than acouple seconds. However, most devices that process and display360-degree videos are equipped with video buffers having much longerdelays than the short-term time horizons of conventional systems thatperform viewport-based adaptive streaming. For instance, it is notuncommon for a client device that displays 360-degree videos to includevideo buffers having 10-15 seconds worth of storage. Moreover,conventional systems that are limited to predicting user viewports forshort-term time horizons do not scale to long-term horizons, since theseconventional systems usually rely just on physical movements of a user,and often model these movements using second-order statistics whichsimply do not include the information needed for long-term time horizonscorresponding to the delays of video buffers in client devices.

Hence, conventional systems that perform viewport-based adaptivestreaming are not efficient because when a video buffer with a long-termtime horizon (e.g., 10-15 seconds) is used, these conventional systemssuffer from quality degradation as the user moves, due to the poorperformance of short-term (e.g., a few seconds) based predictionalgorithms. Conversely, when a video buffer with a short-term timehorizon (e.g., 2-3 seconds) is used, the short-term based predictionalgorithms may be effective at predicting a user viewport for theshort-term time horizon, but delivery of the 360-degree video is moresusceptible to bandwidth fluctuations that cause video freezes and otherquality degradations. Accordingly, these conventional systems yield poorviewing experiences for users.

SUMMARY

Techniques and systems are described for trajectory-based viewportprediction for 360-degree videos. A video system obtains trajectories ofangles as related to a user's viewing angle, and determines the user'sviewport of the 360-degree video over time. For instance, thetrajectories of the angles may include trajectories for a yaw angle, apitch angle, and a roll angle of a user's head recorded as the userviews the 360-degree video for users who have previously viewed the360-degree video. The video system clusters the trajectories of anglesinto trajectory clusters based on a mutual distance between pairs oftrajectories, and determines for each cluster a trend trajectory thatrepresents the trajectories of the trajectory cluster (e.g., an averagetrajectory for the trajectory cluster) and a score threshold thatrepresents the mutual distances for the pairs of trajectories of thetrajectory cluster. When a new user views the 360-degree video, thevideo system compares trajectories of angles of the new user recordedduring a time frame of the 360-degree video to the trend trajectories,and selects trend trajectories for a yaw angle, a pitch angle, and aroll angle for the user based on the comparison and the scorethresholds.

Using the selected trend trajectories for yaw, pitch, and roll angles,the video system predicts viewports of the 360-degree video for the userfor future times (e.g., later times than the time frame of the360-degree video used for the comparison). Hence, the video systempredicts a user's viewport of a 360-degree video based on patterns ofpast viewing behavior of the 360-degree video, e.g., how other usersviewed the 360-degree video. Accordingly, the video system canaccurately predict a user's viewport for long-term time horizons (e.g.,10-15 seconds) that correspond to a device's video buffer delay, so thatthe 360-degree video can be efficiently delivered to a device and viewedwithout undesirable transitions in display quality as the user changeshis or her viewport of the 360-degree video.

This Summary introduces a selection of concepts in a simplified formthat are further described below in the Detailed Description. As such,this Summary is not intended to identify essential features of theclaimed subject matter, nor is it intended to be used as an aid indetermining the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different instances in thedescription and the figures may indicate similar or identical items.Entities represented in the figures may be indicative of one or moreentities and thus reference may be made interchangeably to single orplural forms of the entities in the discussion.

FIG. 1 illustrates a digital medium environment in an exampleimplementation that is operable to employ techniques described herein.

FIG. 2 illustrates an example system usable for trajectory-basedviewport prediction for 360-degree videos in accordance with one or moreaspects of the disclosure.

FIG. 3 illustrates a flow diagram depicting an example procedure inaccordance with one or more aspects of the disclosure.

FIG. 4 illustrates a flow diagram depicting an example procedure inaccordance with one or more aspects of the disclosure.

FIG. 5 illustrates a flow diagram depicting an example procedure inaccordance with one or more aspects of the disclosure.

FIG. 6 illustrates example performance measures in accordance with oneor more aspects of the disclosure.

FIG. 7 illustrates an example system including various components of anexample device that can be implemented as any type of computing deviceas described and/or utilized with reference to FIGS. 1-6 to implementaspects of the techniques described herein.

DETAILED DESCRIPTION

Overview

A 360-degree video (e.g., an immersive video or spherical video)includes multiple viewing directions for the 360-degree video, and canbe used in virtual reality, gaming, and any playback situation where aviewer can control his or her viewing direction and viewport, which isthe part of the 360-degree video viewed by a user at a given time. Forinstance, when viewing a 360-degree video that allows a user to immersethemselves in the 360-degree video using virtual reality, such as in avirtual reality environment of a video game, the viewport correspondingto the user changes over time as the user moves within the virtualreality environment and views different portions of the 360-degreevideo. Consequently, conventional systems may perform viewport-basedadaptive streaming in which a future viewport for a user is predicted,and this future viewport and a current viewport for the user aredelivered (e.g., to a client device over a network) at a higher qualitythan other portions of the 360-degree video that are delivered in alower quality format to reduce bandwidth requirements.

However, these conventional systems predict future viewports overshort-term time horizons, e.g., less than two seconds—far less than thedelays of video buffers typically found on devices that process anddisplay videos, e.g., 10-15 seconds for most client devices.Unfortunately, these conventional systems do not scale to long-termhorizons corresponding to the delays of video buffers. For instance,these conventional systems may rely on physical movements of a userwithout considering how other users viewed the content of the 360-degreevideo, and often use second-order statistics which simply do not includethe information needed for long-term time horizons. When a video bufferwith a long-term time horizon is used, these conventional systems sufferfrom quality degradation as the user moves due to the poor performanceof short-term based prediction algorithms. When a video buffer with ashort-term time horizon is used, delivery of the 360-degree video issusceptible to bandwidth fluctuations that cause video freezes andquality degradations, even if the short-term based prediction algorithmsare effective at predicting a user viewport for the short-term timehorizon. Hence, conventional systems that perform viewport-basedadaptive streaming are inefficient and result in poor viewingexperiences for the user.

Accordingly, this disclosure describes systems, devices, and techniquesfor trajectory-based viewport prediction for 360-degree videos. A videosystem predicts a user viewport for a 360-degree video at a future timebased on trajectories of angles that determine viewports for the360-degree video at earlier times than the future time. Angles mayinclude a yaw angle, a pitch angle, and a roll angle for a user, such asbased on a user's head, eyes, virtual reality device (e.g., head-mountedvirtual-reality goggles), and the like, that are used to determine aviewport for the user.

The video system obtains trajectories of angles, such as a trajectory ofyaw angles, a trajectory of pitch angles, a trajectory of roll angles, atrajectory that jointly represents two or more angles, or combinationsthereof, for a plurality of users for a 360-degree video. The angles aresampled at time instances of the 360-degree video and correspond toviewports of the 360-degree video at the time instances for a pluralityof viewers of the 360-degree video, such as users who have previouslyviewed the 360-degree video. The video system exploits the observationthat many users consume a given 360-degree video in similar ways. Hence,the video system clusters the angle trajectories into trajectoryclusters, and determines trend trajectories (e.g., average trajectories)and score thresholds for each trajectory cluster. When a new user viewsthe 360-degree video for a time period, the video system can match thenew user's angles collected over the time period of the 360-degree videoto one or more trajectory clusters based on the trend trajectories andscore thresholds, and predict a viewport for the new user at a futuretime relative to the time period from the trajectory clusters that matchthe new user's angles.

The video system can cluster the trajectories of angles into trajectoryclusters in any suitable way, including trajectory clusters for yawangle, trajectory clusters for pitch angle, trajectory clusters for rollangle, and trajectory clusters for a joint angle that jointly representstwo or more angles. Hence, the video system can process yaw, pitch, androll angles independently and cluster the trajectories of angles intotrajectory clusters separately for the yaw, pitch, and roll angles.Additionally or alternatively, the video system can process yaw, pitch,and roll angles jointly by processing a joint angle that represents twoor more of the yaw, pitch, and roll angles, and cluster trajectories ofjoint angles into trajectory clusters. Trajectory clusters includetrajectories of angles deemed by the video system to be similar. Forinstance, pairs of trajectories belonging to a trajectory cluster mayhave affinity scores above a threshold affinity score, and the affinityscore for a pair of trajectories may be determined from a mutualdistance between the pair of trajectories.

The video system determines a score threshold for each trajectorycluster identified by the video system. In one example, the video systemdetermines a score threshold for a trajectory cluster based on themutual distances of pairs of trajectories that belong to the trajectorycluster. For instance, the video system may determine a maximum mutualdistance for pairs of trajectories that belong to the trajectory cluster(e.g., the mutual distance for the pair of trajectories that arefarthest from each other among the pairs of trajectories belonging tothe trajectory cluster). The video system may determine the scorethreshold for the trajectory cluster from an affinity score based on themaximum mutual distance for the trajectory cluster. The video systemuses the score thresholds for the trajectory clusters to determine if atrajectory of a new user's angles belongs to a trajectory cluster. Forinstance, the video system may require that a user trajectory (e.g., atrajectory of user angles) and the trend trajectory for the trajectorycluster have an affinity score determined from the mutual distancebetween the user trajectory and the trend trajectory that is greaterthan the score threshold for the trajectory cluster.

The video system can determine a trend trajectory that represents thetrajectories of a trajectory cluster in any suitable way. In oneexample, the video system breaks the 360-degree video into timeintervals (e.g., equally-spaced time intervals), and determines, foreach time interval, polynomial coefficients of a polynomial functionthat is fitted to the trajectories of the trajectory cluster over thetime interval. The video system forms a union over the time intervals ofthe polynomial functions having the polynomial coefficients to determinethe trend trajectory for each trajectory cluster. Hence, a trendtrajectory can be represented as a piecewise polynomial. The videosystem can fit polynomial coefficients of a polynomial function to thetrajectories of the trajectory cluster in any suitable way, such as byselecting the polynomial coefficients to minimize a difference functionbetween the polynomial and a trajectory of angles over all trajectoriesof the trajectory cluster. In one example, the difference functionincludes a mean squared error between the polynomial and a trajectory ofangles over all trajectories of the trajectory cluster. Additionally oralternatively, the difference function may be minimized subject to aboundary constraint on the polynomial functions at boundaries of thetime intervals, to guarantee continuity across the time intervals forthe trend trajectory.

The video system uses the trend trajectories and score thresholds forthe trajectory clusters to predict a viewport for a user from usertrajectories of angles for the user collected over a time period of the360-degree video. The user trajectories can include yaw angles, pitchangles, roll angles, or combinations thereof, and determine userviewports of the 360-degree video during the time period. To predict aviewport for the user at a later time (e.g., a future time) relative tothe time period, the video system determines affinity scores between thetrend trajectories of the trajectory clusters and the user trajectoriesover the time period, and selects at least one trend trajectory based oncomparing the affinity scores to the score thresholds. For instance, ifthe affinity score for a trend trajectory of a trajectory cluster and auser trajectory is greater than the score threshold for the trajectorycluster, then the video system may determine that the user trajectorybelongs to the trajectory cluster. In one example, the video systemselects a first trend trajectory for a yaw angle, a second trendtrajectory for a pitch angle, and a third trend trajectory for a rollangle based on comparing the affinity scores for the trend trajectoriesand user trajectories to the score thresholds of the trajectoryclusters. The video system predicts a user viewport of the 360-degreevideo for a later time than the time period based on the first trendtrajectory, the second trend trajectory, and the third trend trajectory,such as by evaluating the polynomial functions for the first, second,and third trend trajectories at the later time to determine the userviewport for the later time.

In one example, the video system predicts a user viewport for a viewing(e.g., display or playback) of a 360-degree video from trendtrajectories and score thresholds for trajectory clusters thatcorrespond to different viewings of the 360-degree video than theviewing of the 360-degree video. For instance, the video system may bepartially implemented by a server that clusters trajectories of anglesfor users who have previously viewed the 360-degree video (e.g., ahistory of viewings of the 360-degree video). At a later time when a newuser views the 360-degree video on a client device, the server maydeliver to the client device the trend trajectories and score thresholdsfor the trajectory clusters based on data from the history of viewingsof the 360-degree video.

Additionally or alternatively, the video system can predict a userviewport for a display of a 360-degree video from trend trajectories andscore thresholds for trajectory clusters that correspond to a samedisplay (e.g., viewing or exposing) of the 360-degree video for whichthe user viewport is predicted. For instance, when a new user views the360-degree video on a client device, such as during a live event withmultiple simultaneous viewers of the live event, or as part of aninteractive and immersive video game with multiple simultaneous usersplaying the video game, the video system implemented on the clientdevice may obtain trajectories of angles for users who are currentlywatching the live event or playing the video game with the new user. Thevideo system may cluster the trajectories of angles for the users whoare currently watching the live event or playing the video game anddetermine trend trajectories and score thresholds for the trajectoryclusters. Since users may consume portions of the live event or thevideo game in different orders and prior to the new user, the videosystem may predict a viewport for the new user from the trendtrajectories and score thresholds for the trajectory clusters thatinclude trajectories for the users who are currently viewing the liveevent or playing the video game with the new user. Hence, the videosystem may predict or update a prediction of a viewport of a 360-degreevideo for a user based on most-recently available angle trajectories,including angle trajectories for users who are consuming the 360-degreevideo simultaneously with the user, such as watching a live event orplaying a video game concurrently with the user.

Accordingly, the video system predicts a user's viewport of a 360-degreevideo based on patterns of past viewing behavior of the 360-degreevideo, e.g., how other users viewed the 360-degree video, rather thanrelying on methods that do not adequately capture the information neededto predict viewports for long-term time horizons, such as second-orderstatistics or physical models of a user's movement without regard to howother users viewed the content of the 360-degree video. Hence, the videosystem can accurately predict a user's viewport for long-term timehorizons (e.g., 10-15 seconds in the future) that correspond to adevice's video buffer delay, so that the 360-degree video can beefficiently delivered and viewed without undesirable transitions inquality as the user changes the viewport of the 360-degree video.

In the following discussion an example digital medium environment isdescribed that may employ the techniques described herein. Exampleimplementation details and procedures are then described which may beperformed in the example digital medium environment as well as otherenvironments. Consequently, performance of the example procedures is notlimited to the example environment and the example environment is notlimited to performance of the example procedures.

Example Digital Medium Environment

FIG. 1 is an illustration of a digital medium environment 100 in anexample implementation that is operable to employ techniques describedherein. As used herein, the term “digital medium environment” refers tothe various computing devices and resources that can be utilized toimplement the techniques described herein. The illustrated digitalmedium environment 100 includes a user 102 having computing device 104and computing device 106. Computing device 104 is depicted as a pair ofgoggles (e.g., virtual reality goggles), and computing device 106 isdepicted as a smart phone. Computing devices 104 and 106 can include anysuitable type of computing device, such as a mobile phone, tablet,laptop computer, desktop computer, gaming device, goggles, glasses,camera, digital assistant, echo device, image editor, non-linear editor,digital audio workstation, copier, scanner, client computing device, andthe like. Hence, computing devices 104 and 106 may range from fullresource devices with substantial memory and processor resources (e.g.,personal computers, game consoles) to a low-resource device with limitedmemory or processing resources (e.g., mobile devices).

Computing devices 104 and 106 are illustrated as separate computingdevices in FIG. 1 for clarity. In one example, computing devices 104 and106 are included in a same computing device. Notably, computing devices104 and 106 can include any suitable number of computing devices, suchas one or more computing devices, (e.g., a smart phone connected to atablet). Furthermore, discussion of one computing device of one ofcomputing devices 104 and 106 is not limited to that one computingdevice, but generally applies to each of the computing devices 104 and106.

In one example, computing devices 104 and 106 are representative of oneor a plurality of different devices connected to a network that performoperations “over the cloud” as further described in relation to FIG. 7.Additionally or alternatively, computing device 104 can becommunicatively coupled to computing device 106, such as with a lowpower wireless communication standard (e.g., a Bluetooth® protocol).Hence, an asset (e.g., digital image, video, text, drawing, artwork,document, file, and the like) generated, processed, edited, or stored onone device (e.g., a tablet of computing device 106) can be communicatedto, and displayed on and processed by another device (e.g., virtualreality goggles of computing device 104).

Various types of input devices and input instrumentalities can be usedto provide input to computing devices 104 and 106. For example,computing devices 104 and 106 can recognize input as being a mouseinput, stylus input, touch input, input provided through a natural userinterface, and the like. In one example, computing devices 104 and 106may display a 360-degree video, such as in a virtual realityenvironment, and include inputs to interact with the virtual realityenvironment, such as to facilitate a user moving within the virtualreality environment to change the viewport of the 360-degree video(e.g., the user's viewing perspective of the 360-degree video).

In this example of FIG. 1, computing device 104 displays a 360-degreevideo 108, which includes viewport 110 that corresponds to a currentviewport of the 360-degree video 108 for the user 102 (e.g., the portionof the 360-degree video 108 that is currently being viewed by the user102). The 360-degree video 108 may be any suitable size in any suitabledimension. In one example, the 360-degree video 108 spans 360 degreesalong at least one axis, such as a horizontal axis. Additionally oralternatively, the 360-degree video 108 may span 360 degrees in multipleaxes, such as including a spherical display format in which a user canchange his or her viewport 360 degrees in any direction. In one example,the 360-degree video 108 spans less than 360 degrees in at least oneaxis. For instance, the 360-degree video 108 may include a panoramicvideo spanning 180 degrees, in which only a portion of the 180 degreespan is viewable at any one time.

While the user 102 views the 360-degree video 108, computing device 104determines angles 112 for the user 102 that are used to determine aviewport of the 360-degree video 108, such as viewport 110. Forinstance, computing device 104 may include a gyroscope that measures ayaw angle 114, a pitch angle 116, and a roll angle 118 in any suitablecoordinate system, such as a coordinate system for the user's head, acoordinate system for the user's eyes, a coordinate system for gogglesor a head-mounted display of the computing device 104, and the like. Theyaw angle 114, the pitch angle 116, and the roll angle 118 are used todetermine a viewport of the 360-degree video 108 because they correspondto a viewing direction of the 360-degree video 108 for the user 102.Computing device 104 determines values of the yaw angle 114, the pitchangle 116, and the roll angle 118 over time for the user 102, and canstore these values as trajectories for the angles, such as a firsttrajectory including values of the yaw angle 114, a second trajectoryincluding values of the pitch angle 116, and a third trajectoryincluding values of the roll angle 118. The values of the angles may besampled at time instances of the 360-degree video 108 (e.g., based on atimeline of the 360-degree video 108). Accordingly, the trajectories ofthe angles 112 represent a trajectory of the viewport 110 for the user102 as the user changes his or her viewing direction of the 360-degreevideo 108, such as when the user 102 is immersed in a virtual realityenvironment represented by the 360-degree video 108 and moves within theenvironment.

Computing device 104 includes video system 120 that predicts a futureviewport of the 360-degree video 108 for user 102 based on thetrajectories of the angles 112. For instance, the video system 120obtains trend trajectories and score thresholds for trajectory clusters,such as clusters of angle trajectories for users who have previouslyviewed the 360-degree video 108. The video system 120 compares thetrajectories of the angles 112 for the user 102 to the trendtrajectories, and based on the comparisons and the score thresholds,selects a first trend trajectory for the yaw angle 114, a second trendtrajectory for the pitch angle 116, and a third trend trajectory for theroll angle 118 to represent the movement of the user 102 while viewingthe 360-degree video 108.

The video system 120 evaluates the selected first, second, and thirdtrend trajectories at a future time (e.g., a later time than the timeperiod of the trajectories of the angles 112 for the user 102) topredict future viewport 122 of the 360-degree video 108 for the user102. Because the video system 120 selects the first, second, and thirdtrend trajectories based on users' movements while viewing the360-degree video 108 itself, and since most users tend to consume360-degree videos in similar ways, the video system 120 is able toaccurately predict the future viewport 122 for the user 102 forlong-term time horizons that correspond to typical delays of videobuffers, such as 10-15 seconds, which is a significant improvement overconventional systems that are typically limited to viewport predictionfor short-term time horizons (e.g., less than two seconds). Accordingly,the video system 120 can deliver the 360-degree video 108 to the user102 efficiently and without undesirable transitions in the quality ofthe 360-degree video 108 as the user 102 changes the viewport, such asfrom viewport 110 to future viewport 122.

Computing device 106 is also coupled to network 124, whichcommunicatively couples computing device 106 with server 126. Network124 may include a variety of networks, such as the Internet, anintranet, local area network (LAN), wide area network (WAN), personalarea network (PAN), cellular networks, terrestrial networks, satellitenetworks, combinations of networks, and the like, and as such may bewired, wireless, or a combination thereof. For clarity, FIG. 1 does notdepict computing device 104 as being coupled to network 124, thoughcomputing device 104 may also be coupled to network 124 and server 126.

Server 126 may include one or more servers or service providers thatprovide services, resources, assets, or combinations thereof tocomputing devices 104 and 106, such as 360-degree videos. Services,resources, or assets may be made available to video system 120, videosupport system 128, or combinations thereof, and stored at assets 130 ofserver 126. Hence, 360-degree video 108 can include any suitable360-degree video stored at assets 130 of server 126 and delivered to aclient device, such as the computing devices 104 and 106.

Server 126 includes video support system 128 configurable to receivesignals from one or both of computing devices 104 and 106, process thereceived signals, and send the processed signals to one or both ofcomputing devices 104 and 106 to support trajectory-based viewportprediction for 360-degree videos. For instance, computing device 106 mayobtain user angle trajectories (e.g., a yaw angle trajectory, a pitchangle trajectory, and a roll angle trajectory) for user 102, andcommunicate them to server 126. Server 126, using video support system128, may select trend trajectories for yaw angle, pitch angle, and rollangle based on comparing the user angle trajectories received fromcomputing device 106 to trend trajectories for trajectory clusterscorresponding to previous viewings of the 360-degree video 108. Server126 may then send the selected trend trajectories for yaw angle, pitchangle, and roll angle back to computing device 106, which can predict afuture viewport for the user 102 corresponding to a later time with thevideo system 120. Accordingly, the video support system 128 of server126 can include a copy of the video system 120. In one example,computing device 106 sends a request for content of the 360-degree video108 corresponding to the future viewport to server 126, which inresponse delivers the content of the 360-degree video 108 correspondingto the future viewport to computing device 106 at a higher quality(e.g., encoded at a higher bit rate) than other portions of the360-degree video 108, to support viewport-based adaptive streaming.

Computing device 104 includes video system 120 for trajectory-basedviewport prediction for 360-degree videos. The video system 120 includesa display 132, which can be used to display any suitable data used by orassociated with video system 120. In one example, display 132 displays aviewport of a 360-degree video, such as viewport 110 of the 360-degreevideo 108. Portions of the 360-degree video 108 outside the currentviewport may not be displayed in the display 132. As the currentviewport of the 360-degree video 108 is changed over time in response toa user changing his or her viewing direction of the 360-degree video108, the display 132 may change the portion of the 360-degree video 108that is displayed. For instance, at a first time, the display 132 maydisplay content of the 360-degree video 108 corresponding to viewport110, and at a future time (e.g., ten seconds following the first time),the display 132 may display content of the 360-degree video 108corresponding to viewport 122.

The video system 120 also includes processors 134. Processors 134 caninclude any suitable type of processor, such as a graphics processingunit, central processing unit, digital signal processor, processor core,combinations thereof, and the like. Hence, the video system 120 may beimplemented at least partially by executing instructions stored instorage 136 on processors 134. For instance, processors 134 may executeportions of video application 152 (discussed below in more detail).

The video system 120 also includes storage 136, which can be anysuitable type of storage accessible by or contained in the video system120. Storage 136 stores data and provides access to and from memoryincluded in storage 136 for any suitable type of data. For instance,storage 136 includes angle trajectory data 138 including data associatedwith trajectories of angles for viewers of a 360-degree video, such asrepresentations of trajectory clusters (e.g., identification numbers oftrajectory clusters, indications of types of angles of clusters, such asyaw, pitch, roll, or joint angles), angle trajectories belonging totrajectory clusters, identifiers of users corresponding to the angletrajectories, a date of a viewing of a 360-degree video used todetermine angle trajectories, mutual distances for pairs of angletrajectories belonging to trajectory clusters, and affinity scores forpairs of angle trajectories belonging to trajectory clusters. Angletrajectory data 138 may also include trend trajectories for trajectoryclusters (e.g., polynomial coefficients for piecewise polynomials makingup a trend trajectory), a distance measure used to determine trendtrajectories (e.g., minimum mean-squared error, absolute value, anindication of a boundary constraint, etc.), and a number of angletrajectories used to determine trend trajectories (e.g., a number ofangle trajectories of a trajectory cluster used to determine polynomialcoefficients of a trend trajectory for the trajectory cluster). Angletrajectory data 138 may also include score thresholds for trajectoryclusters, a mutual distance used to determine a score threshold (e.g., amaximum mutual distance for pairs of angle trajectories belonging to atrajectory cluster), indications of a pair of angle trajectories used todetermine a score threshold, statistics of score thresholds acrosstrajectory clusters, such as mean, median, mode, variance, etc.,combinations thereof, and the like.

Storage 136 also includes user trajectory data 140 including datarelated to a user viewing a 360-degree video, such as angle trajectoriesthat determine user viewports of the 360-degree video, including a yawangle trajectory, a pitch angle trajectory, a roll angle trajectory, andcombinations thereof. User trajectory data 140 may also include anindication of a location of a device (e.g., a gyroscope) that measuresthe angle trajectories, such as a head of a user, an eye of a user, apair of goggles, etc., and indicators of the 360-degree videocorresponding to the angle trajectories, such as timestamps indicatingtime instances of the 360-degree video, scene identifiers, chapteridentifiers, viewports of the 360-degree video, combinations thereof,and the like.

Storage 136 also includes affinity score data 142 including data relatedto affinity scores for trajectory clusters, such as mutual distancesbetween user trajectories (e.g., trajectories of user angles) and trendtrajectories, and affinity scores determined from the mutual distances.Affinity score data 142 may also include a time period of the 360-degreevideo corresponding to the user trajectories (e.g., time instances forwhich the angles of the user trajectories are sampled), and the like.

Storage 136 also includes selection data 144 including data related todetermining trend trajectories that match user trajectories, such asaffinity scores between user trajectories (e.g., trajectories of userangles) and trend trajectories, differences between the affinity scoresand score thresholds for the trajectory clusters, and selected trendtrajectories (e.g., a first trajectory including values of a yaw angle,a second trajectory including values of a pitch angle, and a thirdtrajectory including values of roll angle) to assign to a user andpredict a future viewport for the user. Selection data 144 may alsoinclude indications of whether or not the selected trend trajectoriescomply with a selection constraint, such as requiring the affinity scorebetween a user trajectory and the trend trajectory to be greater thanthe score threshold for the trajectory cluster represented by the trendtrajectory. For instance, when no trend trajectory satisfies theselection constraint, the video system 120 may select a trend trajectorythat is closest to a user trajectory based on the affinity score for thetrend trajectory and the user trajectory, despite the affinity scorebeing less than the score threshold for the trend trajectory (e.g., forthe trajectory cluster represented by the trend trajectory).

Storage 136 also includes viewport data 146 including data related toviewports of a 360-degree video, such as a predicted viewport (e.g., aviewport for a user predicted by the video system 120) and indicationsof the trend trajectories used to predict the viewport. Viewport data146 may also include data related to a time for which the video system120 predicts the viewport, such as a time horizon (e.g., 10 seconds)from a current time, a percentage of storage of a video buffercorresponding to the time horizon, combinations thereof, and the like.In one example, viewport data 146 includes data related to a viewport,such as content of the 360-degree video for the viewport, an indicatorof quality for content of a viewport, such as encoder rate, and thelike.

Furthermore, the video system 120 includes transceiver module 148.Transceiver module 148 is representative of functionality configured totransmit and receive data using any suitable type and number ofcommunication protocols. For instance, data within video system 120 maybe transmitted to server 126 with transceiver module 148. Furthermore,data can be received from server 126 with transceiver module 148.Transceiver module 148 can also transmit and receive data betweencomputing devices, such as between computing device 104 and computingdevice 106. In one example, transceiver module 148 includes a low powerwireless communication standard (e.g., a Bluetooth® protocol) forcommunicating data between computing devices.

The video system 120 also includes video gallery module 150, which isrepresentative of functionality configured to obtain and manage videos,including 360-degree videos. Hence, video gallery module 150 may usetransceiver module 148 to obtain any suitable data for a 360-degreevideo from any suitable source, including obtaining 360-degree videosfrom a server, such as server 126, computing device 106, or combinationsthereof. Data regarding 360-degree videos obtained by video gallerymodule 150, such as content of 360-degree videos, viewports of360-degree videos, encoder rates for content of 360-degree videos, andthe like can be stored in storage 136 and made available to modules ofthe video system 120.

The video system 120 also includes video application 152. The videoapplication 152 includes angle trajectory module 154, which includescluster module 156, score threshold module 158, and trend trajectorymodule 160. The video application 152 also includes user trajectorymodule 162, affinity score module 164, trajectory selection module 166,and viewport prediction module 168. These modules work in conjunctionwith each other to facilitate trajectory-based viewport prediction for360-degree videos.

Angle trajectory module 154 is representative of functionalityconfigured to determine score thresholds and trend trajectories fortrajectory clusters. For instance, cluster module 156 clusters angletrajectories into trajectory clusters, score threshold module 158determines score thresholds for the trajectory clusters, and trendtrajectory module 160 determines trend trajectories for the trajectoryclusters that represent the trajectories belonging to the trajectoryclusters. Angle trajectory module 154 can determine score thresholds andtrend trajectories for trajectory clusters on any suitable device. Inone example, angle trajectory module 154 is implemented by a server,such as server 126, and the server provides the score thresholds andtrend trajectories for trajectory clusters to a client device (e.g.,computing device 104 or computing device 106). Additionally oralternatively, the angle trajectory module 154 can be implemented by aclient device, such as computing device 104 or computing device 106.

Angle trajectory module 154 can determine score thresholds and trendtrajectories for trajectory clusters at any suitable time to be used bythe video system 120. For instance, a server may provide the scorethresholds and trend trajectories for trajectory clusters to computingdevice 106 periodically (e.g., updated every 24 hours or weekly), inresponse to the user 102 enabling the 360-degree video 108 (e.g., whenthe user 102 begins viewing the 360-degree video 108), during a viewingof the 360-degree video 108 (e.g., after the user 102 has begunconsuming the 360-degree video 108), combinations thereof, and the like.Additionally or alternatively, a client device may generate the scorethresholds and trend trajectories for trajectory clusters periodically,or in response to the user 102 enabling the 360-degree video 108.

In one example, angle trajectory module 154 generates the scorethresholds and trend trajectories for trajectory clusters during adisplay of the 360-degree video 108 (e.g., after the user 102 has begunconsuming the 360-degree video 108). In this case, the score thresholdsand the trend trajectories may be based on angle trajectories of userswho are consuming the 360-degree video 108 at the same time as the user102 (e.g., a same viewing of the 360-degree video 108, such as in amulti-player video game). For instance, the 360-degree video 108 may bebroken into chapters, and the chapters may be consumed by users in aninteractive fashion based on user selections, such as a user's locationwithin a virtual reality environment of a video game. Hence, thechapters may be consumed in different orders by different users of the360-degree video 108. Accordingly, some users may view some chapters ofthe 360-degree video 108 prior to the user 102, but still during a sameplaying of the 360-degree video 108, so that the user movements andangle trajectories may be used by the video system 120 to determinescore thresholds and trend trajectories to predict a viewport for theuser 102 based on the chapter of the 360-degree video 108 being consumedby the user 102.

Cluster module 156 is representative of functionality configured tocluster trajectories of angles into trajectory clusters. In one example,cluster module 156 clusters trajectories separately for different typesof angles, such as by clustering trajectories of yaw angles intotrajectory clusters for yaw angles, clustering trajectories of pitchangles into trajectory clusters for pitch angles, and clusteringtrajectories of roll angles into trajectory clusters for roll angles.

Cluster module 156 can cluster trajectories of angles into trajectoryclusters in any suitable way. In one example, cluster module 156determines mutual distances between pairs of angle trajectories,affinity scores from the mutual distances, and angle trajectoriesbelonging to the trajectory clusters from the affinity scores. Clustermodule 156 can use any suitable distance measure to determine mutualdistances between pairs of angle trajectories. Let P=[p₁ p₂ p₃ . . . ]and Q=[q₁ q₂ q₃ . . . ] represent two trajectories of a type of angle(e.g., a yaw angle), so that p_(i) and q_(i) each denote a yaw angle fora different user, and the subscript i denotes a sample value (e.g., atime instance of a 360-degree video). In one example, cluster module 156determines a mutual distance D(P,Q) between the pair of trajectories Pand Q according to

${D\left( {P,Q} \right)} = {\underset{p \in P}{\overset{\alpha}{ord}}\; d}$${d} = {\min\limits_{q \in {N{({{C{({p,Q})}},Q})}}}{d\left( {p,q} \right)}}$

where C(p,Q) maps a point p∈P to a corresponding point of Q at a samerelative position (e.g., sample value or time instance of the 360-degreevideo), N(q,Q) maps the point q∈Q to the set of neighboring points of Qin the time interval [t_(q)−T_(l); t_(q)+T_(l)] for time instance t_(q)of the point q and tunable parameter T_(l) (e.g., two seconds), andd(p,q) denotes the distance between the points p and q (e.g., theabsolute value of the difference between the angles represented by p andq). Cluster module 156 determines the quantity d

for each point in P, and the mutual distance D(P,Q) is determined fromthe value of d

larger than a percentage α of all values of d

. For instance, for the example where d

=[0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0] and α=80%, the mutualdistance is 0.8. By conditioning the mutual distance based on thepercentage α, cluster module 156 discards outlier values of d

. In one example, the percentage α is set to 95%. Generally, the mutualdistance is not symmetric, so that D(P,Q)≠D(Q,P). Hence, a mutualdistance between the pair of trajectories P and Q can include bothmeasures D(P,Q) and D(Q,P).

Cluster module 156 determines the mutual distance, including both D(P,Q)and D(Q,P), between the pair of trajectories P and Q for alltrajectories to be clustered together (e.g., for trajectories of yawangles separately from trajectories of pitch angles and separately fromtrajectories of roll angles). Based on the mutual distances betweenpairs of trajectories P and Q, the cluster module 156 determines anaffinity score K(P,Q) between the pair of trajectories P and Q for alltrajectories to be clustered together. In one example, the clustermodule 156 determines an affinity score K(P,Q) according to

${{K\left( {P,\ Q} \right)} = {e^{- \frac{{D{({P,Q})}}{D{({Q,P})}}}{2\sigma^{2}}}{\forall P}}},{\forall Q}$

where σ is a tunable scaling parameter that scales the affinity score.In one example, σ is set to a value of ten.

The cluster module 156 includes a spectral clustering algorithm, such asa spectral clustering algorithm as described for vehicle trajectories in“Clustering of Vehicle Trajectories” in IEEE Transactions on IntelligentSystems, 11(3):647-657, September 2010, the disclosure of which isincorporated herein by reference in its entirety. The cluster module 156provides the affinity scores K(P,Q) for all P and Q as inputs to thespectral clustering algorithm, and the spectral clustering algorithmclusters trajectories that are similar based on the affinity scores intotrajectory clusters. For instance, all pairs of a trajectory clusteridentified by cluster module 156 may have an affinity score greater thana threshold affinity score.

Trajectory clusters determined by cluster module 156, along with anysuitable information, such as mutual distances, affinity scores, numbersof trajectory clusters, numbers of angle trajectories belonging totrajectory clusters, identifiers of trajectories (e.g., trajectoryidentification numbers), identifiers of trajectory clusters that eachtrajectory belongs (e.g., cluster identification numbers), parametersused to cluster angle trajectories into trajectory clusters, such aspercentage a, tunable parameter T_(l), and scaling parameter σ,combinations thereof, and the like, used by or calculated by clustermodule 156 are stored in angle trajectory data 138 of storage 136 andmade available to modules of video application 152. In one example,trajectory clusters determined by cluster module 156 are provided totrend trajectory module 160. Additionally or alternatively, clustermodule 156 provides mutual distances of pairs of trajectories intrajectory clusters to score threshold module 158.

Score threshold module 158 is representative of functionality configuredto determine score thresholds for trajectory clusters, such as thetrajectory clusters identified by cluster module 156. A score thresholddetermined by score threshold module 158 can be used by video system 120to determine if a trajectory (e.g., a user trajectory obtained during aviewing of a 360-degree video) belongs to a trajectory cluster, such asby comparing a distance measure between the user trajectory and a trendtrajectory for the trajectory cluster to the score threshold for thetrajectory cluster.

Score threshold module 158 can determine score thresholds for trajectoryclusters in any suitable way. In one example, score threshold module 158determines a score threshold for each trajectory cluster from theminimum affinity score for pairs of trajectories belonging to thetrajectory cluster. For instance, score threshold module 158 may set thescore threshold for a trajectory cluster based on the minimum affinityscore among pairs of trajectories belonging to the trajectory cluster,such as equal to the minimum affinity score, a scaled version of theminimum affinity score (e.g., 110% of the minimum affinity score), andthe like. Hence, the score threshold can represent a minimum affinityscore that a pair of trajectories must have to belong to the trajectorycluster.

Additionally or alternatively, since cluster module 156 determines anaffinity score for a pair of trajectories based on the mutual distancebetween the pair of trajectories, score threshold module 158 candetermine a score threshold for each trajectory cluster from mutualdistances between pairs of trajectories belonging to a trajectorycluster. For instance, score threshold module 158 may determine a scorethreshold for a trajectory cluster from a maximum mutual distance amongthe mutual distances of pairs of trajectories belonging to thetrajectory cluster (e.g., the maximum mutual distance corresponds to apair of trajectories that are farthest apart from one another among thepairs of trajectories belonging to the trajectory cluster). Scorethreshold module 158 may determine a maximum mutual distance in anysuitable way, such as according to

$\max\limits_{P,Q}\left( {{D\left( {P,Q} \right)} + {D\left( {Q,P} \right)}} \right)$

for pairs of trajectories P, Q belonging to the trajectory cluster. Inone example, score threshold module 158 sets the score threshold for atrajectory cluster equal to the maximum mutual distance among the mutualdistances of pairs of trajectories belonging to the trajectory cluster.Additionally or alternatively, score threshold module 158 can set thescore threshold for a trajectory cluster to a scaled version of themaximum mutual distance (e.g., 90% of the maximum mutual distance).Hence, the score threshold can represent a maximum mutual distance thata pair of trajectories can have to belong to the trajectory cluster.

Score thresholds determined by score threshold module 158, along withany suitable information, such as mutual distances, affinity scores,scale factors used to determine the score thresholds, statistics of thescore thresholds across the trajectory clusters (e.g., mean, median,mode, standard deviation, etc.), combinations thereof, and the like,used by or calculated by score threshold module 158 are stored in angletrajectory data 138 of storage 136 and made available to modules ofvideo application 152. In one example, score thresholds determined byscore threshold module 158 are provided to trajectory selection module166.

Trend trajectory module 160 is representative of functionalityconfigured to determine trend trajectories for trajectory clusters, suchas the trajectory clusters identified by cluster module 156. The trendtrajectories determined by trend trajectory module 160 represent thetrajectories belonging to the trajectory clusters. For instance, thetrend trajectories can be an average trajectory for a trajectorycluster, one trajectory of a trajectory cluster, a combination oftrajectories of a trajectory cluster, or any suitable trajectory thatrepresents the trajectories of a trajectory cluster.

Trend trajectory module 160 can determine trend trajectories fortrajectory clusters in any suitable way. In one example, trendtrajectory module 160 selects one trajectory from among trajectories ofa trajectory cluster and approximates the one trajectory by a function,such as a polynomial, a spline of polynomials, piecewise connectedpolynomials, a sum of basis functions, combinations thereof, and thelike. For instance, trend trajectory module 160 may break up the onetrajectory into segments and fit a function to the one trajectory overeach segment. To fit a function to the one trajectory, trend trajectorymodule 160 may minimize a cost function (e.g., mean-squared error) overparameters of the function (e.g., polynomial coefficients) subject to aboundary constraint at the segment boundaries.

By selecting one of the trajectories of a trajectory cluster todetermine a trend trajectory, rather than multiple trajectories, theprocessing requirements needed to determine trend trajectories may bereduced. This reduction in processing requirements can be an advantagein some situations, such as when trend trajectory module 160 is neededto determine the trend trajectories quickly (e.g., responsive to auser-request to display a 360-degree video so that perceptible delay tothe user is minimized), or during a display of a 360-degree video (e.g.,when a user is playing a video game with a 360-degree video environment,the user's viewport can be predicted based on updated trend trajectoriesthat reflect a most recent set of data, including data for users of thevideo game who are concurrently playing the video game).

In one example, trend trajectory module 160 determines trendtrajectories for trajectory clusters based on multiple trajectories inthe trajectory clusters, such as all the trajectories belonging to atrajectory cluster. For instance, trend trajectory module 160 can dividea 360-degree video into intervals (e.g., equally-spaced time intervalsfor the 360-degree video, for chapters of the 360-degree video, forscenes of the 360-degree video, combinations thereof, and the like.). Inone example, the intervals include equally-spaced time intervals of the360-degree video (e.g., three second intervals). For each trajectorycluster determined by cluster module 156, trend trajectory module 160can determine a function r_(i)(t), where subscript i denotes a timeinterval of the 360-degree video. The trend trajectory R^(C) for thetrajectory cluster C is the union of the functions over the timeintervals, or R^(C)=∪_(i)r_(i)(t). Trend trajectory module 160 candetermine trend trajectories for trajectory clusters by fitting thefunction r_(i)(t) to multiple trajectories of the trajectory clusters,such as in a least-squares sense.

Trend trajectory module 160 can use any suitable function r_(i)(t) toconstruct the trend trajectories, such as polynomials, exponentials,trigonometric functions, wavelets, etc. In one example, trend trajectorymodule 160 represents each function r_(i)(t) as an N^(th) orderpolynomial, or

${r_{i}(t)} = {\sum\limits_{j = 0}^{N}{\lambda_{j}t^{j\mspace{11mu}}{\forall{t \in \left\lbrack {T_{i;}T_{i + 1}} \right\rbrack}}}}$

where λ_(j) are polynomial coefficients. The order of the polynomial canbe set to any suitable order. In one example, the order is set to seven(e.g., N=7).

To determine the polynomial coefficients so that the trend trajectoriesrepresent the trajectories of a trajectory cluster, trend trajectorymodule 160 can fit the functions r_(i)(t) to the trajectories of thetrajectory cluster. In one example, trend trajectory module 160 fits thefunctions r_(i)(t) to the trajectories based on a distance measurebetween the trend trajectory and the trajectories in the cluster, suchas a minimum mean-squared error distance measure. For instance, trendtrajectory module 160 can determine the polynomial coefficients tominimize a mean-squared error cost function subject to a boundaryconstraint, or

$\min\limits_{\lambda}{\sum\limits_{P \in C}{\sum\limits_{p \in {P{\lbrack{T_{i,}T_{i + 1}}\rbrack}}}{{p - {r_{i}\left( t_{p} \right)}}}^{2}}}$s.t.  r_(i)(T_(i)) = r_(i − 1)(T_(i))

where t_(p) is the time instance of angle p, P[T_(i),T_(i+1)] indicatesall points of P in the interval [T_(i),T_(i+1)], and P∈C denotesmultiple trajectories of the trajectory cluster C (e.g., alltrajectories of the trajectory cluster C). The constraintr_(i)(T_(i))=r_(i−1)(T_(i)) enforces a boundary condition thatguarantees continuity across time intervals for the trend trajectoryR^(C). Trend trajectory module 160 can determine polynomial coefficientsfor each time interval to determine a trend trajectory for eachtrajectory cluster identified by cluster module 156.

Trend trajectories determined by trend trajectory module 160, along withany suitable information, such as polynomial coefficients, an indicationof a number of trajectories of a cluster used to determine the trendtrajectory for the cluster (e.g., one, all, or some but not all of thetrajectories), values of a cost function (e.g., a minimum mean-squarederror used to determine polynomial coefficients), a time durationbetween samples of angles making up a trajectory of a trajectorycluster, combinations thereof, and the like, used by or calculated bytrend trajectory module 160 are stored in angle trajectory data 138 ofstorage 136 and made available to modules of video application 152. Inone example, trend trajectories determined by trend trajectory module160 are provided to affinity score module 164 and trajectory selectionmodule 166.

User trajectory module 162 is representative of functionality configuredto obtain, during a display of a 360-degree video, user trajectories ofuser angles for a time period of the 360-degree video being displayed.The user angles correspond to user viewports of the 360-degree videoduring the time period of the 360-degree video being displayed. Forinstance, user trajectory module 162 may obtain user trajectories ofuser angles, including a trajectory of yaw angles, a trajectory of rollangles, and a trajectory of pitch angles collected for a time periodwhile a user is viewing the 360-degree video. The angles 112 in FIG. 1are examples of user angles in user trajectories obtained by usertrajectory module 162. For any given time instance of the time period,the user angles can be used to determine a user viewport during the timeperiod. For instance, for any given time instance of the time period,the user angles correspond to a content portion of the 360-degree videoconsumed by the user at the given time instance, such as a portion ofcontent in viewport 110.

User trajectory module 162 can obtain the user trajectories in anysuitable way. For instance, user trajectory module 162 may automaticallyrecord the user trajectories responsive to the 360-degree video 108being displayed, such as when a user 102 enables a viewing device (e.g.,virtual-reality goggles) to display the 360-degree video 108 and aviewport of the 360-degree video 108 is displayed, such as a firstviewport of the 360-degree video 108 displayed to the user 102.Additionally or alternatively, user trajectory module 162 may record theuser trajectories responsive to a user selection to enable recording,such as a “record user angles now” button on a display device, e.g., ahead-mounted display. In one example, a user may select to disregardpreviously-recorded user trajectories, such as by selecting a “recordover” button on a head-mounted display that erases previously-recordeduser trajectories and begins recording new user trajectories whenselected. Hence, the user may disregard old or unreliable usertrajectories and facilitate the video system 120 to better predict afuture viewport for the user based on more reliable user trajectoriesthat more accurately reflect the user's movements and viewports, thusimproving the delivery and playback of the 360-degree video 108 (e.g.,by reducing the transitions in quality as the user moves and changes theviewport).

User trajectories of user angles obtained by user trajectory module 162can include timestamps for each user angle, such as a time valueindicating a time on a timeline of a 360-degree video, a sample number,a time value indicating a time on a time line of a chapter or scene ofthe 360-degree video, a chapter number, a scene number, combinationsthereof, and the like. Hence, a timestamp for a user angle can associatethe user angle with a specific time of a 360-degree video, specificcontent of a 360-degree video, a playback sequence of a 360-degreevideo, combinations thereof, and the like.

User trajectory module 162 can obtain user trajectories that includeangles sampled at any suitable rate. In one example, the sampling rateof the user angles in the user trajectories is based on the 360-degreevideo being displayed. For instance, the sampling rate may be determinedfrom a rate of the 360-degree video, such as derived from a frame rateof the 360-degree video, or set so that a prescribed number of samplesof the user angles are recorded for a chapter or scene of the 360-degreevideo. In one example, the sampling rate of the user angles in the usertrajectories is user-selectable. For instance, a user may select a rateadjuster control via video system 120, such as via a user interfaceexposed by display 132.

User trajectories determined by user trajectory module 162, along withany suitable information, such as a sampling rate of the user angles inthe user trajectories, indicators of a type of user angle of the usertrajectories (e.g., indicators of yaw angles, pitch angles, and rollangles), timestamps of user angles, combinations thereof, and the like,used by or calculated by user trajectory module 162 are stored in usertrajectory data 140 of storage 136 and made available to modules ofvideo application 152. In one example, user trajectories obtained byuser trajectory module 162 are provided to affinity score module 164.

Affinity score module 164 is representative of functionality configuredto determine affinity scores for trajectory clusters based on usertrajectories and trend trajectories for the trajectory clusters.Affinity score module 164 can determine affinity scores based on thetrend trajectories evaluated for times within the time period used tocollect the user trajectories. For instance, for each trajectory clusterfor a type of angle, such as a yaw angle, affinity score module 164determines an affinity score for the trajectory cluster by computing theaffinity score between the trend trajectory for the trajectory clusterand a user trajectory for the type of angle. Since the user angles ofthe user trajectory are collected over a time period, affinity scoremodule 164 evaluates the trend trajectory over this time period tocompute the affinity score.

Let user trajectory U represent a user trajectory of any type of userangles, such as a trajectory of yaw angles or a trajectory of pitchangles, collected over the time period [0;T_(n)]. For instance, the usertrajectory may be recorded by user trajectory module 162 for the firstT_(n) seconds of a 360-degree video. In one example, the affinity scoremodule 164 processes the user trajectories for the different types ofangles separately, such as by first determining affinity scores for yawangles, followed by determining affinity scores for pitch angles,followed by determining affinity scores for roll angles.

The affinity score module 164 determines an affinity score between theuser trajectory U and each of the trend trajectories representingtrajectory clusters for the type of angle (e.g., yaw angles). Hence, theaffinity score module 164 determines an affinity score K(U,R^(C)[0;T_(n)]) based on the mutual distance between the user trajectoryand the trend trajectory as described above. Here, R^(C)[0;T_(n)]denotes the trend trajectory of trajectory cluster C evaluated for timeinstances within the time period [0;T_(n)].

Accordingly, for user trajectories obtained by user trajectory module162, affinity score module 164 can determine a respective affinity scorefor each trajectory cluster clustered by cluster module 156 based on thetrend trajectories for the trajectory clusters and the type of anglesincluded in the trend trajectories. The video system 120 uses theaffinity scores to match the user trajectories to the trajectoryclusters (e.g., to determine which trajectory cluster, if any, a usertrajectory may belong to).

Affinity scores determined by affinity score module 164, along with anysuitable information, such as mutual distances between user trajectoriesand trend trajectories, statistics of affinity scores across trajectoryclusters, such as mean, median, mode, variance, maximum, minimum, etc.,a time period of a 360-degree video for which an affinity score isdetermined (e.g., a time period of user angles included in a usertrajectory), combinations thereof, and the like, used by or calculatedby affinity score module 164 are stored in affinity score data 142 ofstorage 136 and made available to modules of video application 152. Inone example, affinity scores determined by affinity score module 164 areprovided to trajectory selection module 166.

Trajectory selection module 166 is representative of functionalityconfigured to select trend trajectories based on the affinity scoresdetermined by affinity score module 164 and the score thresholdsdetermined by the score threshold module 158. Trajectory selectionmodule 166 selects trend trajectories for trajectory clusters that matchuser trajectories. For instance, when trajectory selection module 166selects a trend trajectory, the trajectory selection module 166determines that a user trajectory belongs to the trajectory clusterrepresented by the trend trajectory.

In one example, trajectory selection module 166 selects a first trendtrajectory for a yaw angle, a second trend trajectory for a pitch angle,and a third trend trajectory for a roll angle from the trendtrajectories based on the affinity scores and the score thresholds. Forinstance, among the trend trajectories representing trajectory clustersfor yaw angles, trajectory selection module 166 selects the trendtrajectory with the highest affinity score as the first trendtrajectory. Among the trend trajectories representing trajectoryclusters for pitch angles, trajectory selection module 166 selects thetrend trajectory with the highest affinity score as the second trendtrajectory. Among the trend trajectories representing trajectoryclusters for roll angles, trajectory selection module 166 selects thetrend trajectory with the highest affinity score as the third trendtrajectory.

Additionally or alternatively, trajectory selection module 166 canselect a trend trajectory based on the score threshold of the trajectorycluster represented by the trend trajectory. For instance, trajectoryselection module 166 may require that the affinity score between thetrend trajectory and the user trajectory be greater than the scorethreshold of the trajectory cluster represented by the trend trajectoryto select the trend trajectory. If this selection constraint based onthe score threshold is not satisfied for one or more of the yaw, pitch,and roll angles, the trajectory selection module 166 may not select atrend trajectory representing the trajectory cluster for the one or moreangles. For instance, the trajectory selection module 166 may not matchthe user trajectory to any trajectory cluster for an angle, such as ayaw angle. In this case, the trajectory selection module 166 may reportthat the user trajectory cannot be matched to an available trajectorycluster for the yaw angle. Additionally or alternatively, the trajectoryselection module 166 may report that the user trajectory can be matchedto an available trajectory cluster for pitch and roll angles, but notfor the yaw angle. In one example, if the selection constraint cannot besatisfied for each of the yaw, pitch, and roll angles (e.g., theaffinity scores are less than the score thresholds), then trajectoryselection module 166 declares the user to be non-predictable and doesnot select any trend trajectory. For instance, the trajectory selectionmodule 166 may cause a message of user non-predictability to bedisplayed on display 132, such as “unable to predict a viewport for thisuser”. Additionally or alternatively, the trajectory selection module166 may request that new user trajectories are obtained, such ascorresponding to a different time period (e.g., a longer time periodthan that used for the user trajectories).

In one example, the trajectory selection module 166 may be configured tobypass the selection constraint based on the score threshold. Forinstance, rather than not select any trend trajectory for an angle type,such as yaw angle, trajectory selection module 166 may select a trendtrajectory that is closest to a user trajectory based on the affinityscore between the trend trajectory and the user trajectory for theangle, such as the trend trajectory having a highest affinity score withthe user trajectory, despite the affinity score being less than thescore threshold. Hence, the selection constraint based on the scorethreshold may be bypassed for some or all angles, such as one or more ofyaw angle, pitch angle, or roll angle.

Trend trajectories determined by trajectory selection module 166, alongwith any suitable information, such as affinity scores, scorethresholds, comparisons of affinity scores and score thresholds,indications of whether a selection constraint is satisfied for selectedtrend trajectories, combinations thereof, and the like, used by orcalculated by trajectory selection module 166 are stored in selectiondata 144 of storage 136 and made available to modules of videoapplication 152. In one example, trend trajectories selected bytrajectory selection module 166 are provided to viewport predictionmodule 168.

Viewport prediction module 168 is representative of functionalityconfigured to predict viewports based on the trend trajectories selectedby trajectory selection module 166. Viewport prediction module 168 canpredict a user viewport (e.g., a viewport for user 102) at a later timethan the time period for which the user trajectories of user angles areobtained by user trajectory module 162 (e.g., a future time). In oneexample, viewport prediction module 168 predicts a user viewport of the360-degree video for a later time than the time period based on a firsttrend trajectory for a yaw angle, a second trend trajectory for a pitchangle, and a third trend trajectory for a roll angle selected bytrajectory selection module 166.

Viewport prediction module 168 can predict a user viewport based ontrend trajectories in any suitable way. In one example, viewportprediction module 168 evaluates the trend trajectories at a later timeto determine a predicted yaw angle, a predicted pitch angle, and apredicted roll angle that determine the predicted user viewport. Forinstance, let R_(φ)*, R_(ω)*, and R_(θ)* represent trend trajectoriesselected by trajectory selection module 166 for yaw angle φ, pitch angleω, and roll angle θ, respectively, based on a user trajectory havinguser angles over the time period [0;T_(n)]. To predict yaw, pitch, androll angles at a future time T_(n)+T_(h), trajectory selection module166 evaluates R_(φ)*, R_(ω)*, and R_(θ)* at future time T_(n)+T_(h), or

φ_(T) _(n) _(+T) _(h) =R ₁₀₀*(T _(n) +T _(h))

ω_(T) _(n) _(+T) _(h) =R _(ω)*(T _(n) +T _(h))

θ_(T) _(n) _(+T) _(h) =R _(θ)*(T _(n) +T _(h)).

These predicted yaw, pitch, and roll angles at future time T_(n)+T_(h)determine a viewing direction for the user at the future time, and hencecan be used to determine a user viewport at the future time for the360-degree video (e.g., a predicted user viewport). Viewport 122 is anexample of a user viewport predicted by viewport prediction module 168for the 360-degree video 108.

The time horizon T_(h) can be a long-term time horizon, such as equal toa delay of a video buffer of computing device 104 or computing device106 (e.g., 10-15 seconds). Accordingly, the time horizon T_(h) is notlimited to the short-term time horizons of conventional systems that areoften less than two seconds, and typically just hundreds ofmilliseconds. Hence, the video system 120 can predict user viewports forlong-term time horizons so that the 360-degree video can be efficientlydelivered to a device and viewed without undesirable transitions inquality as the user changes the viewport of the 360-degree video.

User viewports predicted by viewport prediction module 168, along withany suitable information, such as a time horizon T_(h), predicted yawangles, predicted pitch angles, predicted roll angles, combinationsthereof, and the like, used by or calculated by viewport predictionmodule 168 are stored in viewport data 146 of storage 136 and madeavailable to modules of video application 152. In one example, a userviewport predicted by viewport prediction module 168 is communicated toa server (e.g., server 126) to request content of a 360-degree videocorresponding to the user viewport at a higher quality than otherportions of the 360-degree video in a viewport-based adaptive streamingdelivery of the 360-degree video.

Example Video System

FIG. 2 illustrates an example system 200 usable for trajectory-basedviewport prediction for 360-degree videos in accordance with one or moreaspects of the disclosure. In this implementation, system 200 includesthe modules of video application 152 as described in FIG. 1, e.g., angletrajectory module 154, which includes cluster module 156, scorethreshold module 158, and trend trajectory module 160. The system 200also includes user trajectory module 162, affinity score module 164,trajectory selection module 166, and viewport prediction module 168 ofthe video application 152. System 200 is one example of video system 120that can be constructed using the modules of video application 152. Forinstance, signals can be redefined, and modules can be modified,combined, divided, added, or removed to form a modified system, withoutaltering the functionality of system 200. Accordingly, such modifiedsystems are considered to be within the scope of the disclosure.

Furthermore, for simplicity, system 200 is limited to the modules ofvideo application 152 and a description of some of their interconnects.System 200 can, however, include any suitable signals and communicationsbetween modules omitted for simplicity. Such signals may include systemclocks, counters, timestamps of a 360-degree video (e.g., a timeline,chapter indicator, scene number, etc.), angle type indicators,trajectory cluster designators (e.g., angle types of a trajectorycluster, cluster identification numbers, etc.), reset signals, and thelike.

System 200 can be implemented on any suitable device or devices. In oneexample, system 200 is implemented on one computing device (e.g.,computing device 104 or computing device 106 in FIG. 1). In anotherexample, system 200 is implemented on more than one computing device.For instance, parts of system 200 can be implemented by a firstcomputing device, such as computing device 104 in FIG. 1, and otherparts of system 200 can be implemented by an additional computingdevice, such as computing device 106. In one example, a serverimplements parts of system 200, such as server 126 in FIG. 1. A servercan be remote, e.g., because it is not collocated with another computingdevice, such as computing device 106. A server may be configured toreceive signals of system 200 from a computing device (e.g., computingdevice 104), process the received signals, such as with video supportsystem 128, and transmit results of the processing back to the computingdevice. Hence, the video support system 128 of server 126 in FIG. 1 mayinclude system 200. In one example, a server implements cluster module156, trend trajectory module 160, and score threshold module 158, and aclient computing device (e.g., computing device 106) implements usertrajectory module 162, affinity score module 164, trajectory selectionmodule 166, and viewport prediction module 168.

In one example, the modules of system 200 are executed during a displayof a 360-degree video (e.g., while a user is viewing the 360-degreevideo). Additionally or alternatively, some of the modules of system 200may be executed during a display of a 360-degree video, and othermodules of system 200 can be executed prior to the display of the360-degree video. For instance, cluster module 156, trend trajectorymodule 160, and score threshold module 158 may be implemented prior tothe display of a 360-degree video to generate trajectory clusters, trendtrajectories, and score thresholds, respectively, which may be usedduring the display of the 360-degree video by user trajectory module162, affinity score module 164, trajectory selection module 166, andviewport prediction module 168 to predict a future viewport for a userviewing the 360-degree video.

In the example in FIG. 2, a 360-degree video 202 depicts a landscapescene, and viewports 204 and 206 depict viewports for a user during atime period of the 360-degree video 202. For instance, during a timeperiod the 360-degree video 202 is played (e.g., delivered or served), auser has viewed the content of the 360-degree video 202 in viewports 204and 206. Viewport 208 is a viewport predicted by system 200 for the userviewing the 360-degree video 202, and corresponds to a predicted viewingdirection of the 360-degree video 202 for the user at a later time(e.g., a future time) relative to the time period over which the userviews the 360-degree video 202 via the viewports 204 and 206.

Cluster module 156 obtains angle trajectories for the 360-degree video202, such as trajectories of yaw angles, trajectories of pitch angles,and trajectories of roll angles. These angle trajectories determineviewports of the 360-degree video 202 for viewers of the 360-degreevideo 202, such as users who have previously viewed the 360-degree video202. For instance, the angles can be sampled at time instances of the360-degree video 202, and the angles at the time instances cancorrespond to viewports of the 360-degree video at the time instances.In one example, a server maintains a database of angle trajectories for360-degree video (e.g., server 126 in FIG. 1), and system 200 obtainsthe angle trajectories from the server, such as periodically (e.g., oncea month), in response to a new user viewing the 360-degree video 202(e.g., a new user presently viewing the 360-degree video 202 or a newuser whose user angles have been added to the database of angletrajectories).

Cluster module 156 clusters the angle trajectories into trajectoryclusters based on mutual distances between pairs of the trajectories.For instance, cluster module 156 may include a spectral clusteringalgorithm that receives as input mutual distances or affinity scoresdetermined from the mutual distances and generates trajectory clustersthat include trajectories of angles. To cluster the angle trajectoriesinto trajectory clusters, cluster module 156 may require that pairs oftrajectories belonging to a trajectory cluster satisfy a distanceconstraint, such as their mutual distances being less than a distancethreshold or their affinity scores being greater than a thresholdaffinity score. In one example, cluster module 156 clusters trajectoriesfor yaw angles, pitch angles, and roll angles separately, so that atrajectory cluster determined by cluster module 156 is for a given typeof angle (e.g., yaw angles). Hence, cluster module 156 may assign anidentification to each trajectory cluster that identifies and describesthe trajectory cluster, such as including a type of angle represented bythe trajectory cluster, a number of trajectories in the trajectorycluster, statistics of the mutual distances or affinity scores for pairsof trajectories in the trajectory cluster, user identifiers for thetrajectories in the trajectory cluster, combinations thereof, and thelike. Cluster module 156 provides trajectory clusters to trendtrajectory module 160, including trajectories of the trajectory clustersand any suitable identification of the trajectory clusters. Clustermodule 156 also provides mutual distances of pairs of trajectories foreach trajectory cluster to score threshold module 158. Each mutualdistance between a pair of trajectories P and Q can include mutualdistance measures D(P,Q) and D(Q,P), as described above.

Trend trajectory module 160 receives trajectory clusters from clustermodule 156 and determines, for each trajectory cluster, a respectivetrend trajectory for the trajectory cluster that represents thetrajectories belonging to the trajectory cluster. Trend trajectorymodule 160 can determine trend trajectories for the trajectory clustersin any suitable way. In one example, trend trajectory module 160 dividesthe 360-degree video 202 into time intervals, and for each timeinterval, determines a function that fits one or more of thetrajectories of the trajectory cluster. For instance, trend trajectorymodule 160 may determine a polynomial that matches the trajectories of atrajectory cluster for each time interval by minimizing a distancefunction between the trajectories of the trajectory cluster and thepolynomial over choices of polynomial coefficients, such as amean-squared error distance measure that is subject to a boundaryconstraint to ensure continuity of the trend trajectory at theboundaries of the time intervals. Hence, trend trajectory module 160 maydetermine a trend trajectory as a piecewise polynomial function.

In one example, trend trajectory module 160 determines a trendtrajectory for a trajectory cluster based on multiple trajectories of atrajectory cluster, such as all the trajectories belonging to thetrajectory cluster. Additionally or alternatively, when reducedprocessing time is desired, such as during the viewing of the 360-degreevideo 202 in which system 200 uses the trend trajectories to predictfuture viewport 208, trend trajectory module 160 may determine a trendtrajectory for a trajectory cluster based on one of the trajectories ofa trajectory cluster, such as by selecting a single trajectory andfitting a polynomial function to the single trajectory at each timeinterval of the 360-degree video. Trend trajectory module 160 providestrend trajectories for the trajectory cluster (e.g., a different trendtrajectory for each trajectory cluster) to affinity score module 164 andtrajectory selection module 166.

Score threshold module 158 receives mutual distances from cluster module156. For instance, score threshold module 158 receives, for eachtrajectory cluster identified by cluster module 156, mutual distances ofpairs of trajectories belonging to the trajectory cluster. For eachtrajectory cluster, score threshold module 158 determines a scorethreshold based on the mutual distances of pairs of trajectoriesbelonging to the trajectory cluster.

Score threshold module 158 can determine score thresholds based on themutual distances in any suitable way. In one example, score thresholdmodule 158 determines affinity scores based on the mutual distances asdescribed above, and sets the score threshold for a trajectory clusterto the minimum affinity score among the affinity scores computed fromthe mutual distances for the pairs of trajectories belonging to thetrajectory cluster. Additionally or alternatively, score thresholdmodule 158 can determine, for each trajectory cluster a maximum mutualdistance among the mutual distances of the pairs of trajectoriesbelonging to the trajectory cluster. Score threshold module 158 maycompute an affinity score for this maximum mutual distance, and set thescore threshold for the trajectory cluster to the affinity scorecomputed from the maximum mutual distance. Hence, the score thresholdmodule 158 can determine a score threshold for each trajectory clusterthat represents a minimum affinity score that a pair of trajectoriesmust have to belong to the trajectory cluster, or a maximum mutualdistance that a pair of trajectories can have to belong to thetrajectory cluster. Score threshold module 158 provides score thresholdsfor the trajectory clusters to trajectory selection module 166.

User trajectory module 162 obtains, during a display of the 360-degreevideo 202, user angles for a time period of the 360-degree video 202being displayed (e.g., user trajectories of the user angles). The userangles correspond to viewports of the 360-degree video for a user duringthe time period, including viewport 204 and viewport 206. Usertrajectory module 162 may record a trajectory of yaw angles, atrajectory of pitch angles, and a trajectory of roll angles for a useras the user views the 360-degree video 202 over the time period. Usertrajectory module 162 provides the user trajectories recorded during thetime period to affinity score module 164.

Affinity score module 164 receives user trajectories from usertrajectory module 162 and trend trajectories from trend trajectorymodule 160, and computes affinity scores between the user trajectoriesand the trend trajectories. Affinity score module 164 may computeaffinity scores separately for yaw angles, pitch angles, and rollangles. For instance, affinity score module 164 may compute affinityscores between a user trajectory for a yaw angle and trend trajectoriesthat represent trajectory clusters for yaw angles. Affinity score module164 may also compute affinity scores between a user trajectory for apitch angle and trend trajectories that represent trajectory clustersfor pitch angles. Affinity score module 164 may also compute affinityscores between a user trajectory for a roll angle and trend trajectoriesthat represent trajectory clusters for roll angles.

Affinity score module 164 can compute affinity scores in any suitableway. In one example, for a user trajectory U including angles over thetime period [0;T_(n)] and a trend trajectory R, affinity score module164 computes the mutual distance between the user trajectory and thetrend trajectory evaluated over the time period, D(U, R[0;T_(n)]) andD(R[0;T_(n)], U). Affinity score module 164 determines the affinityscore from the mutual distance computations according to

${K\left( {U,{R\left\lbrack {0;T_{n}} \right\rbrack}} \right)} = e^{- \frac{{D{({U,{R{\lbrack{0;T_{n}}\rbrack}}})}}{D{({{R{\lbrack{0;T_{n}}\rbrack}},U})}}}{2\; \sigma^{2}}}$

where, as described above, σ is a scaling parameter, such as ten.Affinity score module 164 provides a respective affinity score for eachtrajectory cluster to trajectory selection module 166.

Trajectory selection module 166 receives affinity scores for eachtrajectory cluster from affinity score module 164, and score thresholdsfor each trajectory cluster from score threshold module 158, and selectstrend trajectories to represent the movement of the user viewing the360-degree video 202. For instance, trajectory selection module 166 canselect a first trend trajectory for a yaw angle, a second trendtrajectory for a pitch angle, and a third trend trajectory for a rollangle from the trend trajectories based on the affinity scores and thescore thresholds.

The trajectory selection module 166 can select trend trajectories in anysuitable way. In one example, trajectory selection module 166 selectsthe trend trajectories for the different types of angles having thehighest affinity scores (e.g., the trend trajectories corresponding tothe trajectory clusters with the highest affinity scores computed byaffinity score module 164). For instance, among trend trajectories foryaw angles, trajectory selection module 166 can select as the firsttrend trajectory for a yaw angle the trend trajectory corresponding tothe trajectory cluster for yaw angles with a highest affinity score.Among trend trajectories for pitch angles, trajectory selection module166 can select as the second trend trajectory for a pitch angle thetrend trajectory corresponding to the trajectory cluster for pitchangles with a highest affinity score. Among trend trajectories for rollangles, trajectory selection module 166 can select as the third trendtrajectory for a roll angle the trend trajectory corresponding to thetrajectory cluster for roll angles with a highest affinity score.

In one example, trajectory selection module 166 may apply a selectionconstraint in which, for a trend trajectory to be selected by trajectoryselection module 166, the affinity score determined by the affinityscore module 164 for the trajectory cluster represented by the trendtrajectory must be greater than the score threshold for the trajectorycluster determined by the score threshold module 158. If a trendtrajectory cannot be found to satisfy the selection constraint (e.g.,the score threshold is greater than the affinity score for thetrajectory cluster with a highest affinity score), trajectory selectionmodule 166 may report to the user that their user trajectories do notmatch any trajectories on record for the 360-degree video 202.Additionally or alternatively, if a trend trajectory cannot be found tosatisfy the selection constraint, trajectory selection module 166 maybypass the selection constraint and select the user trajectoryrepresenting the trajectory cluster with the highest affinity score. Inone example, trajectory selection module 166 is configured to bypass theselection constraint based on receiving a user input, such as a userinput to override the selection constraint (e.g., an “override” buttonon a head-mounted display for a virtual realty environment of the360-degree video 202 that indicates to predict a user viewport using thebest data available even if a selection constraint cannot be satisfiedwith the available data).

Trajectory selection module 166 provides selected trend trajectories toviewport prediction module 168. In one example, trajectory selectionmodule 166 provides a first trend trajectory for a yaw angle, a secondtrend trajectory for a pitch angle, and a third trend trajectory for aroll angle to viewport prediction module 168.

Viewport prediction module 168 receives trend trajectories fromtrajectory selection module 166 (e.g., trend trajectories that system200 determines are a best fit to the user trajectories over the timeperiod of viewing the 360-degree video 202), and predicts a userviewport for a future time. The future time is a later time than timesof the time period for which the user angles are recorded by usertrajectory module 162. For instance, the future time is a later timethan the times corresponding to the viewports 204 and 206.

Viewport prediction module 168 can predict a user viewport for a futuretime in any suitable way. In one example, viewport prediction module 168evaluates a first trend trajectory for a yaw angle, a second trendtrajectory for a pitch angle, and a third trend trajectory for a rollangle at the future time to determine predicted yaw, pitch, and rollangles, respectively. The predicted angles are used to determine thefuture viewport 208, a predicted user viewport at the future time.

Accordingly, system 200 can accurately predict a user viewport forlong-term time horizons (e.g., 10-15 seconds) that correspond to adevice's video buffer delay, so that the 360-degree video 202 can beefficiently delivered to a device and viewed without undesirabletransitions in quality as the user changes the viewport of the360-degree video 202, such as from viewport 204 to viewport 208. In oneexample, system 200 sends a request for content of the 360-degree video202 corresponding to the future viewport 208, such as request to aserver to deliver the content corresponding to the future viewport 208at a higher quality (e.g., a higher encoder rate) than other portions ofthe 360-degree video 202.

In one example, system 200 separately processes different types ofangles. For instance, system 200 may separate trajectory clusters, trendtrajectories, and user trajectories for yaw angles, pitch angles, androll angles. System 200 may predict a yaw angle for a user viewport fromthe trajectory clusters, trend trajectories, and user trajectories foryaw angles, a pitch angle for a user viewport from the trajectoryclusters, trend trajectories, and user trajectories for pitch angles,and a roll angle for a user viewport from the trajectory clusters, trendtrajectories, and user trajectories for roll angles.

Additionally or alternatively, system 200 can jointly process angles foryaw, pitch, and roll. For instance, system 200 may define a joint angleπ that simultaneously represents two or more angles, such as

$\pi = \begin{bmatrix}\phi \\\omega \\\theta\end{bmatrix}$

where φ denotes a yaw angle, ω denotes a pitch angle, and θ denotes aroll angle. System 200 may determine trajectory clusters for the jointangle π, and match user trajectories of the joint angle to thetrajectory clusters by comparing the user trajectories to trendtrajectories of the trajectory clusters for the joint angle. Once system200 selects a trend trajectory representing the joint angle for a user,the system 200 can predict the user's viewport by evaluating theselected trend trajectory representing the joint angle at a future time.By processing joint angles, system 200 may be able to exploitdependencies between angles and better match user movements to atrajectory cluster, increasing the reliability of system 200 andextending the time horizons over which system 200 can accurately predicta user viewport.

The systems described herein constitute an improvement over conventionalsystems that predict a user viewport based on a user's movement withoutregard to how other users viewed content of the 360-degree video, orthat rely on second-order-statistics that do not capture the informationneeded to accurately predict a user viewport over a long-term timehorizon. In contrast, the systems described herein match a new user'smovement and viewing direction during the display of a 360-degree videoto trajectories of yaw, pitch, and roll angles representing users whohave previously viewed the 360-degree video. Since users tend to consumecontent of a 360-degree video in similar ways, the systems describedherein can accurately predict the new user's viewport at a future timeby evaluating the trajectories of yaw, pitch, and roll angles at thefuture time. Unlike conventional systems, the systems described hereincan predict user viewports for long-term time horizons (e.g., 10-15seconds), and therefore can fill a video buffer with usable content.Accordingly, the systems described herein can be used to efficientlydeliver a 360-degree video with viewport-based adaptive streamingmethods so that the 360-degree video can be viewed without undesirabletransitions in quality as the user changes the viewport of the360-degree video over time.

Example Procedures

FIG. 3 illustrates an example procedure 300 for trajectory-basedviewport prediction for 360-degree videos in accordance with one or moreaspects of the disclosure. Aspects of the procedure may be implementedin hardware, firmware, or software, or a combination thereof. Theprocedure is shown as a set of blocks that specify operations performedby one or more devices and are not necessarily limited to the ordersshown for performing the operations by the respective blocks. In atleast some aspects, the procedure may be performed in a digital mediumenvironment by a suitably configured computing device, such as one ormore of computing device 104, computing device 106, or server 126 ofFIG. 1 that makes use of a video system, such as system 200 or videosystem 120. A video system implementing procedure 300 may be anindependent application that has been installed on the computing device,a service hosted by a service provider that is accessible by thecomputing device, a plug-in module to the computing device, orcombinations thereof.

Trajectories of angles that are sampled at time instances of the360-degree video are received (block 302). In one example, angletrajectory module 154 receives trajectories of angles that are sampledat time instances of the 360-degree video. The angles at the timeinstances correspond to viewports of a 360-degree video at the timeinstances. The angles can include, at each time instance, at least oneof a yaw angle, a pitch angle, or a roll angle. In one example, thetrajectories include separate trajectories for different ones of theangles, such as trajectories for yaw angles, trajectories for pitchangles, and trajectories for roll angles. Additionally or alternatively,the trajectories can include a trajectory for a joint angle thatsimultaneously represents multiple ones of the angles, such as a jointangle representing a yaw angle, a pitch angle, and a roll angle.

In one example, the trajectories include a trajectory for an angledetermined from at least one of a yaw angle, a pitch angle, or a rollangle. For instance, coordinates on a sphere can be represented in anysuitable way based on yaw, pitch, and roll angles, such as by usingquaternions components. Hence, any angle coordinate representationderived from yaw, pitch, and roll angles can be used to determine anangle having a trajectory obtained by angle trajectory module 154.

The trajectories are clustered into trajectory clusters based on mutualdistances between pairs of the trajectories (block 304). In one example,cluster module 156 clusters the trajectories into trajectory clustersbased on mutual distances between pairs of the trajectories. Forinstance, cluster module 156 can cluster the trajectories with aspectral clustering algorithm that receives affinity scores for pairs oftrajectories that are calculated from the mutual distances between thepairs of trajectories.

Score thresholds for the trajectory clusters are determined from themutual distances between the pairs of the trajectories belonging to thetrajectory clusters (block 306). In one example, score threshold module158 determines score thresholds for the trajectory clusters from themutual distances between the pairs of the trajectories belonging to thetrajectory clusters.

In one example, determining the score thresholds for the trajectoryclusters includes determining, for each trajectory cluster, a maximumone of the mutual distances between the pairs of the trajectoriesbelonging to each trajectory cluster. For each trajectory cluster, ascore threshold is generated based on the maximum one of the mutualdistances for the trajectory cluster, such as by computing an affinityscore from the maximum one of the mutual distances.

Trend trajectories are determined for the trajectory clusters (block308). The trend trajectories represent the trajectories belonging to thetrajectory clusters. In one example, trend trajectory module 160determines trend trajectories for the trajectory clusters, the trendtrajectories representing the trajectories belonging to the trajectoryclusters. Determining the trend trajectories for the trajectory clusterscan include determining time intervals of the 360-degree video. For eachtime interval, polynomial coefficients are determined for eachtrajectory cluster based on the angles in the trajectories of thetrajectory cluster during the time interval. Determining the polynomialcoefficients can include minimizing a difference function of the anglesand the polynomial functions over the polynomial coefficients subject toa boundary constraint on the polynomial functions at boundaries of thetime intervals. For each trajectory cluster, a union is formed over thetime intervals of polynomial functions that have the polynomialcoefficients. Additionally or alternatively, trend trajectory module 160can determine trend trajectories for the trajectory clusters bydetermining, for each trajectory cluster of the trajectory clusters, acentroid trajectory (e.g., a mean trajectory), a median trajectory, orcombinations thereof.

A user viewport of the 360-degree video is predicted for a future timeinstance from the score thresholds, angle samples of at least one of thetrend trajectories, and user angles that correspond to the user viewport(block 310). The angle samples and the user angles correspond to thetime instances occurring prior to the future time instance. In oneexample, viewport prediction module 168 predicts a viewport of the360-degree video for a future time instance from trend trajectoriesselected by trajectory selection module 166, and trajectory selectionmodule 166 selects the trend trajectories based on affinity scorescomputed by affinity score module 164 between user trajectories of theuser angles and the trend trajectories. User trajectory module 162 canobtain the user angles (e.g., user trajectories of user angles).

In one example, the user angles and the angles correspond to differentuser-viewings of the 360-degree video. For instance, the user angles maycorrespond to a current viewing of the 360-degree video by a new user,and some of the angles may correspond to previous viewings by differentusers of the 360-degree video. Additionally or alternatively, the userangles and the angles may correspond to a shared-viewing of the360-degree video. For instance, multiple players may be simultaneouslyimmersed in a virtual reality environment of a video game represented bythe 360-degree video. The user angles may correspond to one of themultiple players, and the angles may correspond to other players of themultiple players.

FIG. 4 illustrates an example procedure 400 for trajectory-basedviewport prediction for 360-degree videos in accordance with one or moreaspects of the disclosure. Aspects of the procedure may be implementedin hardware, firmware, or software, or a combination thereof. Theprocedure is shown as a set of blocks that specify operations performedby one or more devices and are not necessarily limited to the ordersshown for performing the operations by the respective blocks. In atleast some aspects, the procedure may be performed in a digital mediumenvironment by a suitably configured computing device, such as one ormore of computing device 104, computing device 106, or server 126 ofFIG. 1 that makes use of a video system, such as system 200 or videosystem 120. A video system implementing procedure 400 may be anindependent application that has been installed on the computing device,a service hosted by a service provider that is accessible by thecomputing device, a plug-in module to the computing device, orcombinations thereof.

Score thresholds and trend trajectories for trajectory clusters areobtained (block 402). The trend trajectories represent angletrajectories clustered into the trajectory clusters for yaw angles,pitch angles, and roll angles that are sampled at time instances of a360-degree video. The angles correspond to viewports of the 360-degreevideo at the time instances. In one example, angle trajectory module 154obtains score thresholds and trend trajectories for trajectory clusters,the trend trajectories representing angle trajectories clustered intothe trajectory clusters for yaw angles, pitch angles, and roll anglesthat are sampled at time instances of a 360-degree video and correspondto viewports of the 360-degree video at the time instances.

Angle trajectory module 154 can obtain score thresholds and trendtrajectories in any suitable way. In one example, the score thresholdsand trend trajectories are pre-computed, and angle trajectory module 154obtains the pre-computed score thresholds and trend trajectories from aserver, such as prior to a display of the 360-degree video. Additionallyor alternatively, obtaining the score thresholds and the trendtrajectories can include generating the score thresholds and the trendtrajectories. For instance, cluster module 156 may cluster the angletrajectories into trajectory clusters, score threshold module 158 maydetermine score thresholds for the trajectory clusters, and trendtrajectory module 160 may determine the trend trajectories for thetrajectory clusters, such as during a display of the 360-degree video tobe used during the display of the 360-degree video.

During a display of the 360-degree video, user trajectories of userangles for a time period of the 360-degree video being displayed areobtained (block 404). The user angles correspond to user viewports ofthe 360-degree video. In one example, user trajectory module 162obtains, during a display of the 360-degree video, user trajectories ofuser angles for a time period of the 360-degree video being displayed,the user angles corresponding to user viewports of the 360-degree video.

Affinity scores for the trajectory clusters are determined based on theuser trajectories and the trend trajectories (block 406). The trendtrajectories are evaluated for times of the time instances within thetime period. In one example, affinity score module 164 determinesaffinity scores for the trajectory clusters based on the usertrajectories and the trend trajectories, the trend trajectoriesevaluated for times of the time instances within the time period. Theaffinity score module 164 can be configured to determine the affinityscores based on mutual distances between the user trajectories and thetrend trajectories.

A first trend trajectory for a yaw angle, a second trend trajectory fora pitch angle, and a third trend trajectory for a roll angle areselected from the trend trajectories based on the affinity scores andthe score thresholds (block 408). In one example, trajectory selectionmodule 166 selects a first trend trajectory for a yaw angle, a secondtrend trajectory for a pitch angle, and a third trend trajectory for aroll angle from the trend trajectories based on the affinity scores andthe score thresholds. The trajectory selection module 166 can beconfigured to select at least one of the first trend trajectory, thesecond trend trajectory, or the third trend trajectory based on theaffinity scores being greater than the score thresholds for thetrajectory clusters corresponding to the first trend trajectory, thesecond trend trajectory, or the third trend trajectory.

In one example, the trajectory selection module 166 can be configured toselect one trajectory of the first trend trajectory, the second trendtrajectory, or the third trend trajectory based on the user trajectorybeing closer to the one trajectory than other trajectories of the trendtrajectories. Additionally or alternatively, the trajectory selectionmodule 166 can be further configured to select the one trajectory basedon an affinity score for a trajectory cluster that includes the onetrajectory not satisfying a constraint based on comparing the affinityscore to a score threshold for the trajectory cluster.

A user viewport of the 360-degree video is predicted for a later timethan the time period (block 410). The user viewport can be predictedbased on the first trend trajectory, the second trend trajectory, andthe third trend trajectory. In one example, viewport prediction module168 predicts a user viewport of the 360-degree video for a later timethan the time period based on the first trend trajectory, the secondtrend trajectory, and the third trend trajectory. The viewportprediction module 168 can be configured to predict the user viewport byevaluating the first trend trajectory, the second trend trajectory, andthe third trend trajectory at the later time. In one example, theviewport prediction module 168 is configured to send a request forcontent of the 360-degree video corresponding to the user viewport, suchas a request to a server for the content.

In one example, the viewport prediction module 168 is configured todetermine a time horizon based on the time period and the later time.For instance, the viewport prediction module 168 may determine the timehorizon from a difference in time between the later time and a time ofthe time period, such as a current time or a latest time of the timeperiod. The viewport prediction module 168 may determine a percentage ofstorage of a video buffer corresponding to the time horizon. Forinstance, viewport prediction module 168 may determine a percentage ofstorage of a video buffer based on the amount of memory needed to storethe 360-degree video over the time horizon.

FIG. 5 illustrates an example procedure 500 for trajectory-basedviewport prediction for 360-degree videos in accordance with one or moreaspects of the disclosure. Aspects of the procedure may be implementedin hardware, firmware, or software, or a combination thereof. Theprocedure is shown as a set of blocks that specify operations performedby one or more devices and are not necessarily limited to the ordersshown for performing the operations by the respective blocks. In atleast some aspects, the procedure may be performed in a digital mediumenvironment by a suitably configured computing device, such as one ormore of computing device 104, computing device 106, or server 126 ofFIG. 1 that makes use of a video system, such as system 200 or videosystem 120. A video system implementing procedure 500 may be anindependent application that has been installed on the computing device,a service hosted by a service provider that is accessible by thecomputing device, a plug-in module to the computing device, orcombinations thereof.

Score thresholds and trend trajectories for trajectory clusters areobtained (block 502). The trend trajectories represent angletrajectories clustered into the trajectory clusters, angles of the angletrajectories corresponding to viewports of a 360-degree video. In oneexample, angle trajectory module 154 obtains score thresholds and trendtrajectories for trajectory clusters, the trend trajectoriesrepresenting angle trajectories clustered into the trajectory clusters,angles of the angle trajectories corresponding to viewports of a360-degree video.

Angle trajectory module 154 can obtain score thresholds and trendtrajectories in any suitable way. In one example, the score thresholdsand trend trajectories are pre-computed, and angle trajectory module 154obtains the pre-computed score thresholds and trend trajectories from aserver. Additionally or alternatively, obtaining the score thresholdsand the trend trajectories can include generating the score thresholdsand the trend trajectories. For instance, cluster module 156 may clusterthe angle trajectories into trajectory clusters, score threshold module158 may determine score thresholds for the trajectory clusters, andtrend trajectory module 160 may determine the trend trajectories for thetrajectory clusters.

During a display of the 360-degree video, user trajectories of userangles for a time period of the 360-degree video being displayed areobtained (block 504). The user angles correspond to user viewports ofthe 360-degree video. In one example, user trajectory module 162obtains, during a display of the 360-degree video, user trajectories ofuser angles for a time period of the 360-degree video being displayed,the user angles corresponding to user viewports of the 360-degree video.

In one example, angle trajectory module 154 obtains the score thresholdsand the trend trajectories responsive to a request for the display ofthe 360-degree video. Additionally or alternatively, angle trajectorymodule 154 can obtain the score thresholds and the trend trajectoriesduring the display of the 360-degree video. In one example, for eachtrajectory cluster, a pair of the angle trajectories belonging to thetrajectory cluster that have a larger mutual distance than other pairsof the angle trajectories belonging to the trajectory cluster aredetermined. Angle trajectory module 154 can determine a score thresholdfor each trajectory cluster based on the pair of the angle trajectoriesbelonging to the trajectory cluster that have the largest mutualdistance.

Affinity scores for the trajectory clusters are determined based on theuser trajectories and the trend trajectories for the time period (block506). In one example, affinity score module 164 determines affinityscores for the trajectory clusters based on the user trajectories andthe trend trajectories for the time period. For instance, the trendtrajectories are evaluated over the time period and used with the usertrajectories to compute affinity scores, as described above.

At least one of the trend trajectories is selected based on the affinityscores and the score thresholds (block 508). In one example, trajectoryselection module 166 selects at least one of the trend trajectoriesbased on the affinity scores and the score thresholds. In one example,trajectory selection module 166 selects a first trend trajectory for ayaw angle, a second trend trajectory for a pitch angle, and a thirdtrend trajectory for a roll angle from the trend trajectories based onthe affinity scores and the score thresholds.

A user viewport of the 360-degree video is determined for a later timethan the time period from the at least one of the trend trajectories(block 510). In one example, viewport prediction module 168 determines auser viewport of the 360-degree video for a later time than the timeperiod from the at least one of the trend trajectories.

In one example, the video system ascertains whether a selectionconstraint is satisfied, such as by comparing the affinity scores andthe score thresholds for the trajectory clusters. For instance, thevideo system may ascertain that a selection constraint is satisfied fora trajectory cluster if an affinity score is greater than the scorethreshold for the trajectory cluster. Responsive to the selectionconstraint not being satisfied, the video system may request new usertrajectories of new user angles that may be matched to trajectoryclusters. When the selection constraint is satisfied, however, the videosystem may select at least one of the trend trajectories and determinethe user viewport.

The procedures described herein constitute an improvement overconventional methods that predict a user viewport based on a user'smovement without regard to how other users viewed content of the360-degree video, or that rely on second-order-statistics that do notcapture the information needed to accurately predict a user viewportover a long-term time horizon. In contrast, the procedures describedherein match a new user's movement during the display of a 360-degreevideo to trajectories of yaw, pitch, and roll angles determined frommovement of users who have previously viewed the 360-degree video. Sinceusers tend to consume content of a 360-degree video in similar ways, theprocedures described herein can accurately predict the new user'sviewport at a future time by evaluating the trajectories of yaw, pitch,and roll angles at the future time. Unlike conventional methods, theprocedures described herein can predict user viewports for long-termtime horizons (e.g., 10-15 seconds), and therefore can fill a videobuffer with usable content. Accordingly, the procedures described hereincan be used to efficiently deliver a 360-degree video withviewport-based adaptive streaming methods so that the 360-degree videocan be viewed without undesirable transitions in quality as the userchanges the viewport of the 360-degree video over time.

Example Results

FIG. 6 illustrates example performance measures 600 in accordance withone or more aspects of the disclosure. Performance measures 600illustrate example results for three systems, including system 200 inFIG. 2, a naïve system (referred to as “fixed angle”) that fixes theangle at time T_(n)+T_(h) to the angle at time T_(n) for time horizonT_(h), and a modified linear regression system (referred to as “linearregression”) as described in “Cub360: Exploiting cross-users behaviorsfor viewport prediction in 360 degree video adaptive streaming” in IEEEInternational Conference on Multimedia and Expo, 2018, by Y. Ban et al.In FIG. 6, results for system 200 are denoted with dark circles, resultsfor the fixed angle system are denoted with dark squares, and resultsfor the linear regression system are denoted with dark triangles, asillustrated in the key 602.

Performance measures 600 show example results determined for 16different 360-degree videos. Each of the 360-degree videos are viewed byup to 61 users, of which 80% are used to determine trajectory clusters,and trend trajectories and score thresholds for the trajectory clusters.The remaining 20% of the viewers are used for viewport prediction by thesystems being compared. Graphs 604, 606, and 608 illustrate thecumulative density function (CDF) of the average viewport overlappercentage (between predicted viewports and actual viewports for theusers) for time horizons of 1 second, 5 seconds, and 10 seconds,respectively, averaged over all the users and the 16 different360-degree videos. In these graphs, for a viewport overlap percentage onthe x-axis, the corresponding point on the y-axis represents thepercentage of users whose viewport overlap is smaller than the viewportoverlap percentage on the x-axis. As an example, if the value on thex-axis for a curve is 0.8 and the corresponding value for the curve onthe y-axis is 0.25, then 75% of the users have a viewport overlappercentage between their predicted viewport and their actual viewportgreater than 80%.

For T_(h)=1 second, the results of graph 604 show that the three systemshave similar performance. However, for longer-term time horizons ofT_(h)=5 seconds and T_(h)=10 seconds, the results of graph 606 and 608show that the system 200 significantly outperforms the fixed anglesystem and the linear regression system. Moreover, the results forsystem 200 are consistent for the different time horizons in graphs 604,606, and 608, indicating that system 200 is accurately able to match auser's movements to other viewers' movements, and that the differentusers tend to consume the 360-degree videos in similar ways.

Illustrations 610 and 612 show box-plots for two of the 360-degreevideos, respectively, for each of the three systems tested for a timehorizon of T_(h)=5 seconds. The box-plots show statistics of theviewport overlap percentage across the users viewing the 360-degreevideos. For instance, in each of the box-plots, the box (or rectangle)bottom corresponds to the 25^(th) percentile (called Q1) and the box topcorresponds to the 75th percentile (called Q3). The dashed line inside abox is the median value. Upper and lower values for ranges of the dataare indicated by upper and lower vertical lines extending from theboxes, respectively, and terminating in horizontal dashes. Thehorizontal dash for the upper vertical line indicates an upper value ofthe range of data and is calculated according to Q3+1.5·(Q3−Q1), and thehorizontal dash for the lower vertical line indicates a lower value ofthe range of data, which is calculated according to Q1−1.5·(Q3−Q1).

Illustrations 610 and 612 show that statistically, system 200outperforms the other two systems significantly. For instance, not onlyis the median performance for system 200 much better than the medianperformance values of the fixed angle system and the linear regressionsystem, but also the distribution across the users indicates higherviewport overlap percentage for system 200 than for the other twosystems.

Example Systems and Devices

FIG. 7 illustrates an example system 700 including an example computingdevice 702 that is representative of one or more computing systems anddevices that can be utilized to implement the various techniquesdescribed herein. This is illustrated through inclusion of video system120, system 200, video application 152, and video support system 128,which operate as described above. Computing device 702 may be, forexample, a user computing device (e.g., computing device 104 orcomputing device 106), or a server device of a service provider, (e.g.,server 126). Furthermore, computing device 702 may include an on-chipsystem, multiple computing devices, combinations thereof, or any othersuitable computing device or computing system. Accordingly, FIG. 7illustrates computing device 702 as one or more of a tablet, a laptopcomputer, a smart phone, smart eye glasses, and a camera, though theseexamples are illustrative and in no way are meant to limit the type ornumber of devices that may be represented by computing device 702.

The example computing device 702 includes a processing system 704, oneor more computer-readable media 706, and one or more I/O interfaces 708that are communicatively coupled to each other. Although not shown,computing device 702 may further include a system bus or other data andcommand transfer system that couples the various components, one toanother. A system bus can include any one or combination of differentbus structures, such as a memory bus or memory controller, a peripheralbus, a universal serial bus, and a processor or local bus that utilizesany of a variety of bus architectures. A variety of other examples arealso contemplated, such as control and data lines.

Processing system 704 is representative of functionality to perform oneor more operations using hardware. Accordingly, processing system 704 isillustrated as including hardware elements 710 that may be configured asprocessors, functional blocks, and so forth. This may includeimplementation in hardware as an application specific integrated circuitor other logic device formed using one or more semiconductors. Hardwareelements 710 are not limited by the materials from which they are formedor the processing mechanisms employed therein. For example, processorsmay be comprised of semiconductor(s) and transistors (e.g., electronicintegrated circuits (ICs)). In such a context, processor-executableinstructions may be electronically-executable instructions. Processors134 in FIG. 1 are an example of processing system 704.

Computer-readable storage media 706 is illustrated as includingmemory/storage 712. Storage 136 in FIG. 1 is an example ofmemory/storage of memory/storage 712. Memory/storage 712 may includevolatile media (such as random access memory (RAM)), nonvolatile media(such as read only memory (ROM), Flash memory, optical disks, magneticdisks, and so forth), or combinations thereof. Memory/storage 712 mayinclude fixed media (e.g., RAM, ROM, a fixed hard drive, and so on) aswell as removable media (e.g., Flash memory, a removable hard drive, anoptical disc, and so forth). Computer-readable media 706 may beconfigured in a variety of other ways as further described below.

Input/output interfaces 708 are representative of functionality to allowa user to enter commands and information to computing device 702, andalso allow information to be presented to the user and other componentsor devices using various input/output devices. Examples of input devicesinclude a keyboard, a cursor control device (e.g., a mouse), amicrophone, an array of microphones, a scanner, touch functionality(e.g., capacitive or other sensors that are configured to detectphysical touch), a camera (e.g., which may employ visible or non-visiblewavelengths such as infrared frequencies to recognize movement asgestures that do not involve touch), and so forth. In one example,computing device 702 includes speech recognition, identification, andsynthesis functionalities, microphones, and speakers that allowcomputing device 702 to communicate with a user in a conversation, e.g.,a user conversation. Accordingly, computing device 702 can recognizeinput as being a mouse input, stylus input, touch input, input providedthrough a natural user interface, and the like. Thus, computing device702 can recognize multiple types of gestures including touch gesturesand gestures provided through a natural user interface.

Examples of output devices include a display device (e.g., a monitor orprojector), speakers, a printer, a network card, tactile-responsedevice, and so forth. In one example, computing device 702 displays a360-degree video, such as a virtual reality environment in which a usermay be immersed and move to change the viewport of the 360-degree video.For instance, input/output interfaces 708 may include a display that candisplay a 360-degree video and can include any suitable type of display,such as a touchscreen, liquid crystal display, plasma display,head-mounted display, projector and screen, and the like. A touchscreencan include any suitable type of touchscreen, such as a capacitivetouchscreen, a resistive touchscreen, a surface acoustic wavetouchscreen, an infrared touchscreen, an optical imaging touchscreen, anacoustic pulse recognition touchscreen, combinations thereof, and thelike. Thus, computing device 702 may be configured in a variety of waysas further described below to support user interaction.

Computing device 702 also includes applications 714. Applications 714are representative of any suitable applications capable of running oncomputing device 702, and may include a web browser which is operable toaccess various kinds of web-based resources (e.g., assets, media clips,images, content, configuration files, services, user profiles, and thelike). Applications 714 include video application 152, as previouslydescribed. Furthermore, applications 714 includes any applicationssupporting video system 120, system 200, and video support system 128.

Various techniques may be described herein in the general context ofsoftware, hardware elements, or program modules. Generally, such modulesinclude routines, programs, objects, elements, components, datastructures, and so forth that perform particular tasks or implementparticular abstract data types. The terms “module,” “functionality,” and“component” as used herein generally represent software, firmware,hardware, or a combination thereof. The features of the techniquesdescribed herein are platform-independent, meaning that the techniquesmay be implemented on a variety of commercial computing platforms havinga variety of processors.

An implementation of the described modules and techniques may be storedon or transmitted across some form of computer-readable media. Thecomputer-readable media may include a variety of media that may beaccessed by computing device 702. By way of example, and not limitation,computer-readable media may include “computer-readable storage media”and “computer-readable signal media.”

“Computer-readable storage media” refers to media, devices, orcombinations thereof that enable persistent or non-transitory storage ofinformation in contrast to mere signal transmission, carrier waves, orsignals per se. Thus, computer-readable storage media does not includesignals per se or signal bearing media. The computer-readable storagemedia includes hardware such as volatile and non-volatile, removable andnon-removable media, storage devices, or combinations thereofimplemented in a method or technology suitable for storage ofinformation such as computer readable instructions, data structures,program modules, logic elements/circuits, or other data. Examples ofcomputer-readable storage media may include, but are not limited to,RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM,digital versatile disks (DVD) or other optical storage, hard disks,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, or other storage device, tangible media, orarticle of manufacture suitable to store the desired information andwhich may be accessed by a computer.

“Computer-readable signal media” refers to a signal-bearing medium thatis configured to transmit instructions to the hardware of the computingdevice 702, such as via a network. Signal media typically may embodycomputer readable instructions, data structures, program modules, orother data in a modulated data signal, such as carrier waves, datasignals, or other transport mechanism. Signal media also include anyinformation delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media include wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared, and other wireless media.

As previously described, hardware elements 710 and computer-readablemedia 706 are representative of modules, programmable device logic,fixed device logic implemented in a hardware form, or combinationsthereof that may be employed in some aspects to implement at least someaspects of the techniques described herein, such as to perform one ormore instructions. Hardware may include components of an integratedcircuit or on-chip system, an application-specific integrated circuit(ASIC), a field-programmable gate array (FPGA), a complex programmablelogic device (CPLD), and other implementations in silicon or otherhardware. In this context, hardware may operate as a processing devicethat performs program tasks defined by instructions, logic embodied bythe hardware, or combinations thereof, as well as a hardware utilized tostore instructions for execution, e.g., the computer-readable storagemedia described previously.

Combinations of the foregoing may also be employed to implement varioustechniques described herein. Accordingly, software, hardware, orexecutable modules may be implemented as one or more instructions, logicembodied on some form of computer-readable storage media or by one ormore hardware elements 710, or combinations thereof. Computing device702 may be configured to implement particular instructions and functionscorresponding to the software and hardware modules. Accordingly,implementation of a module that is executable by computing device 702 assoftware may be achieved at least partially in hardware, e.g., throughuse of computer-readable storage media and hardware elements 710 ofprocessing system 704. The instructions and functions may beexecutable/operable by one or more articles of manufacture (for example,one or more computing devices such as computing device 702 or processingsystems such as processing system 704) to implement techniques, modules,and examples described herein.

The techniques described herein may be supported by variousconfigurations of computing device 702 and are not limited to thespecific examples of the techniques described herein. This functionalitymay also be implemented all or in part through use of a distributedsystem, such as over a “cloud” 716 via a platform 718. Cloud 716includes and is representative of a platform 718 for resources 720.Platform 718 abstracts underlying functionality of hardware (e.g.,servers) and software resources of cloud 716.

Resources 720 may include applications, data, or applications and datathat can be utilized while computer processing is executed on serversthat are remote from computing device 702. Resources 720 can alsoinclude services provided over the Internet, through a subscribernetwork, such as a cellular or Wi-Fi network, or combinations thereof.Generally, resources 720 may be licensed, purchased, or may be madefreely available, (e.g., without authentication, license, oraccount-based access). Resources 720 can include asset store 722, whichstores assets, such as 360-degree videos that may be accessed bycomputing device 702. The resources 720 can include any suitablecombination of services and content, such as an on-line shoppingservice, an image editing service, an artwork drawing service, a webdevelopment and management service, a collaboration service, a socialnetworking service, a messaging service, an advertisement service, agraphics design service, an animation service, an image storage service(including storage of photos, documents, records, files, and the like),a graphics editing service, an asset distribution service, and so forth.Content may include various combinations of assets, including videos,ads, audio, multi-media streams, animations, digital images, digitalartworks, web documents, web pages, applications, device applications,text documents, drawings, presentations, photographs (e.g., stockphotographs), user profiles, user preferences, user data (e.g., imagesstored in an image gallery), maps, computer code, and the like.

Platform 718 may abstract resources and functions to connect computingdevice 702 with other computing devices. Platform 718 may also serve toabstract scaling of resources to provide a corresponding level of scaleto encountered demand for resources 720 that are implemented viaplatform 718. Accordingly, in an interconnected device embodiment,implementation of functionality described herein may be distributedthroughout system 700. For example, the functionality may be implementedin part on computing device 702 as well as via platform 718 thatabstracts the functionality of cloud 716.

CONCLUSION

In one or more implementations, a digital medium environment includes atleast one computing device. Systems, devices, and techniques aredescribed herein for trajectory-based viewport prediction for 360-degreevideos. A video system obtains trajectories of angles of users who havepreviously viewed a 360-degree video. The angles are used to determinethe users' viewports of the 360-degree video, and may includetrajectories for a yaw angle, a pitch angle, and a roll angle of auser's head recorded as the user views the 360-degree video. The videosystem clusters the trajectories of angles into trajectory clustersbased on a mutual distance between pairs of trajectories, and for eachtrajectory cluster determines a trend trajectory and a score threshold.When a new user views the 360-degree video, the video system comparestrajectories of angles of the new user to the trend trajectories, andselects trend trajectories for a yaw angle, a pitch angle, and a rollangle for the user based on the comparison and the score thresholds.Using the selected trend trajectories for yaw angle, pitch angle, androll angle, the video system predicts viewports of the 360-degree videofor the user for future times. Hence, the video system predicts a user'sviewport of a 360-degree video based on patterns of past viewingbehavior of the 360-degree video, e.g., how other users viewed the360-degree video. Accordingly, the video system can accurately predict auser's viewport for long-term time horizons corresponding to videobuffer delays, so that the 360-degree video can be efficiently deliveredand viewed without undesirable transitions in quality as the userchanges the viewport of the 360-degree video.

Although implementations of trajectory-based viewport prediction for360-degree videos have been described in language specific to featuresand/or methods, the appended claims are not necessarily limited to thespecific features or methods described. Rather, the specific featuresand methods are disclosed as example implementations of trajectory-basedviewport prediction for 360-degree videos, and other equivalent featuresand methods are intended to be within the scope of the appended claims.Further, various different examples are described and it is to beappreciated that each described example can be implemented independentlyor in connection with one or more other described examples.

What is claimed is:
 1. In a digital medium environment for viewportprediction of a video, a method implemented by a computing device, themethod comprising: obtaining, by the computing device, trendtrajectories for yaw angles, pitch angles, and roll angles that aresampled at time instances of a video and correspond to viewports of thevideo at the time instances; obtaining, by the computing device during adisplay of the video, user trajectories of user angles for a time periodof the video being displayed, the user angles corresponding to userviewports of the video; determining, by the computing device, affinityscores for the trend trajectories based on the user trajectories;selecting, by the computing device, a first trend trajectory for a yawangle, a second trend trajectory for a pitch angle, and a third trendtrajectory for a roll angle from the trend trajectories based on theaffinity scores; and predicting, by the computing device, a userviewport of the video for a later time than the time period based on thefirst trend trajectory, the second trend trajectory, and the third trendtrajectory.
 2. The method as described in claim 1, wherein thedetermining the affinity scores is based on mutual distances between theuser trajectories and the trend trajectories.
 3. The method as describedin claim 1, wherein the selecting the at least one of the first trendtrajectory, the second trend trajectory, or the third trend trajectoryis based on the affinity scores being greater than the score thresholdsfor trajectory clusters corresponding to the first trend trajectory, thesecond trend trajectory, or the third trend trajectory.
 4. The method asdescribed in claim 1, wherein the selecting includes selecting onetrajectory of the first trend trajectory, the second trend trajectory,or the third trend trajectory based on the user trajectory being closerto the one trajectory than other trajectories of the trend trajectories.5. The method as described in claim 4, wherein the selecting the onetrajectory is based on an affinity score for a trajectory cluster thatincludes the one trajectory not satisfying a constraint based oncomparing the affinity score to a score threshold for the trajectorycluster.
 6. The method as described in claim 1, wherein the predictingincludes: determining a time horizon based on the time period and thelater time; and determining a percentage of storage of a video buffercorresponding to the time horizon.
 7. The method as described in claim1, further comprising sending a request for content of the videocorresponding to the user viewport.
 8. A video system implemented by acomputing device in a digital medium environment, the video systemincluding modules implemented at least partially in hardware of thecomputing device, the video system comprising: an angle trajectorymodule to obtain trend trajectories for yaw angles, pitch angles, androll angles that are sampled at time instances of video and correspondto viewports of the 360-degree video at the time instances; a usertrajectory module to obtain, during a display of the video, usertrajectories of user angles for a time period of the video beingdisplayed, the user angles corresponding to user viewports of the video;an affinity score module to determine affinity scores for the trendtrajectories based on the user trajectories; a trajectory selectionmodule to select a first trend trajectory for a yaw angle, a secondtrend trajectory for a pitch angle, and a third trend trajectory for aroll angle based on the affinity scores; and a viewport predictionmodule to predict a user viewport of the video for a later time than thetime period based on the first trend trajectory, the second trendtrajectory, and the third trend trajectory.
 9. The video system asdescribed in claim 8, wherein the affinity score module is configured todetermine the affinity scores based on mutual distances between the usertrajectories and the trend trajectories.
 10. The video system asdescribed in claim 8, wherein the trajectory selection module isconfigured to select at least one of the first trend trajectory, thesecond trend trajectory, or the third trend trajectory based on theaffinity scores being greater than the score thresholds for trajectoryclusters corresponding to the first trend trajectory, the second trendtrajectory, or the third trend trajectory.
 11. The video system asdescribed in claim 8, wherein the trajectory selection module isconfigured to select one trajectory of the first trend trajectory, thesecond trend trajectory, or the third trend trajectory based on the usertrajectory being closer to the one trajectory than other trajectories ofthe trend trajectories.
 12. The video system as described in claim 11,wherein the trajectory selection module is further configured to selectthe one trajectory based on an affinity score for a trajectory clusterthat includes the one trajectory not satisfying a constraint based oncomparing the affinity score to a score threshold for the trajectorycluster.
 13. The video system as described in claim 8, wherein theviewport prediction module is configured to: determine a time horizonbased on the time period and the later time; and determine a percentageof storage of a video buffer corresponding to the time horizon.
 14. Thevideo system as described in claim 8, wherein the viewport predictionmodule is configured to send a request for content of the 360-degreevideo corresponding to the user viewport.
 17. In a digital mediumenvironment for viewport prediction of a video, a system comprising:means for obtaining trend trajectories for yaw angles, pitch angles, androll angles that are sampled at time instances of a video and correspondto viewports of the video at the time instances; means for obtaining,during a display of the video, user trajectories of user angles for atime period of the video being displayed, the user angles correspondingto user viewports of the video; means for determining affinity scoresfor the trend trajectories based on the user trajectories; means forselecting a first trend trajectory for a yaw angle, a second trendtrajectory for a pitch angle, and a third trend trajectory for a rollangle from the trend trajectories based on the affinity scores; andmeans for predicting a user viewport of the video for a later time thanthe time period based on the first trend trajectory, the second trendtrajectory, and the third trend trajectory.
 18. The system as describedin claim 17, wherein the means for determining the affinity scores isbased on mutual distances between the user trajectories and the trendtrajectories.
 19. The system as described in claim 17, wherein the meansfor selecting the at least one of the first trend trajectory, the secondtrend trajectory, or the third trend trajectory is based on the affinityscores being greater than the score thresholds for trajectory clusterscorresponding to the first trend trajectory, the second trendtrajectory, or the third trend trajectory.
 20. The system as describedin claim 17, wherein the means for selecting includes selecting onetrajectory of the first trend trajectory, the second trend trajectory,or the third trend trajectory based on the user trajectory being closerto the one trajectory than other trajectories of the trend trajectories.